[PLUG] Virtual memory fragmentation and Linux

Tim tim-pdxlug at sentinelchicken.org
Fri Jul 31 18:58:12 UTC 2009


Hello Daniel,

> I'm looking for info on virtual memory fragmentation on Linux. I have 
> come across a few articles indicating that processes on Windows that 
> malloc/free memory all the time will fragment the memory until no 
> contiguous memory is available and the process will receive an out of 
> memory error.
> 
> Although the malloc/free memory call is made at the process level, would 
> Linux face the same limitations, or does the way Linux handles virtual 
> memory alleviate this problem.

I don't have any good references handy, but here's how I understand
it:

malloc(3) and friends are largely implemented in userspace.  Whenever
more memory needs to be allocated, brk(2) or sbrk(2) is called under
the hood to extend the data segment available to malloc.  So from a
virtual memory space perspective, this system is certainly susceptible
to memory fragmentation.  I'm sure malloc tries hard to avoid
fragmentation through smart algorithms and such, but diabolical cases
most certainly exist to diminish the usable contiguous space.  Of
course it's always possible that malloc will actually go through the
effort of defragmenting the data segment if some big chunk of memory
is requested, but surely that's avoided when possible.

Note that there are many different implementations of malloc and more
recently, a lot of tweaking has been done in various settings for
security reasons to help mitigation heap-based buffer overflows.  Some
implementations use mmap(2) under the hood in conjunction with
brk/sbrk.  This allows malloc algorithms to eliminate some internal
fragmentation by allowing external fragmentation in virtual memory
space (i.e. separate mmaped segments need not be mapped contiguously
in virtual memory space).

I did just run across this thread, which may be helpful:
  http://fixunix.com/unix/84604-memory-fragmentation-malloc.html

In the end, the nice thing about malloc is that it's just a library
function in user space.  You can always implement your own version if
some special case of fragmentation is biting you.

HTH,
tim



More information about the PLUG mailing list