[PLUG] Out of space on device

Hal Pomeranz hal at deer-run.com
Sat May 9 13:50:38 UTC 2009


> I ran into a problem on a server I run last night. df was reporting no space
> left and services obviously stopped working. The weird thing is that when I
> checked all the directories on the mount point they didn't add up to the
> space allocated to the partition. It's a 37GB mounted as / with /var /bin
> /boot /sbin /usr /dev /media /proc /opt and .sys
> 
> I removed some un-needed software on the box to get it back up and breathing
> again but I still have no clue how it filled up, so I have no way of knowing
> if it'll happen again. Originally IIRC it was only using about 17GB of space
> as the meat of storage is on /home on a seperate partition.
> 
> Hitting me with the clue stick would be appreciated or any ideas on where to
> look would be great.

I'm guessing you have an "open but unlinked" file someplace.  In other
words, there's a process that's writing a big data file, but that file
was removed.  Since the file has been removed ("unlinked") from the file
system, tools like "du" won't find the space used by this file, which is
why your file system totals don't match up.  However, the operating system
is unable to reclaim the space until all processes that have the file
open have been terminated.

How can you find those processes?  "lsof +L1 | sort -nk7" (as root)
will show you all "files that are open with link count less than 1",
(i.e. the open but unlinked files) and then sort them by size, so you can
find the big ones easier.  Use the PID in the second column to kill
the appropriate processes.  Note that not all open but unlinked
files are necessarily a problem.  It's my experience that programs
like VMware and Wine use open but unlinked files a lot.

-- 
Hal Pomeranz, Founder/CEO      Deer Run Associates      hal at deer-run.com
    Network Connectivity and Security, Systems Management, Training



More information about the PLUG mailing list