[PLUG] Out of space on device

drew wymore drew.wymore at gmail.com
Sat May 9 13:59:21 UTC 2009


On Sat, May 9, 2009 at 6:50 AM, Hal Pomeranz <hal at deer-run.com> wrote:

> > I ran into a problem on a server I run last night. df was reporting no
> space
> > left and services obviously stopped working. The weird thing is that when
> I
> > checked all the directories on the mount point they didn't add up to the
> > space allocated to the partition. It's a 37GB mounted as / with /var /bin
> > /boot /sbin /usr /dev /media /proc /opt and .sys
> >
> > I removed some un-needed software on the box to get it back up and
> breathing
> > again but I still have no clue how it filled up, so I have no way of
> knowing
> > if it'll happen again. Originally IIRC it was only using about 17GB of
> space
> > as the meat of storage is on /home on a seperate partition.
> >
> > Hitting me with the clue stick would be appreciated or any ideas on where
> to
> > look would be great.
>
> I'm guessing you have an "open but unlinked" file someplace.  In other
> words, there's a process that's writing a big data file, but that file
> was removed.  Since the file has been removed ("unlinked") from the file
> system, tools like "du" won't find the space used by this file, which is
> why your file system totals don't match up.  However, the operating system
> is unable to reclaim the space until all processes that have the file
> open have been terminated.
>
> How can you find those processes?  "lsof +L1 | sort -nk7" (as root)
> will show you all "files that are open with link count less than 1",
> (i.e. the open but unlinked files) and then sort them by size, so you can
> find the big ones easier.  Use the PID in the second column to kill
> the appropriate processes.  Note that not all open but unlinked
> files are necessarily a problem.  It's my experience that programs
> like VMware and Wine use open but unlinked files a lot.
>
> --
> Hal Pomeranz, Founder/CEO      Deer Run Associates      hal at deer-run.com
>    Network Connectivity and Security, Systems Management, Training
> _______________________________________________
> PLUG mailing list
> PLUG at lists.pdxlinux.org
> http://lists.pdxlinux.org/mailman/listinfo/plug
>

Thanks Hal,
I found some processes related to Apache and MySQL which aren't a problem.
Nothing else showing up that looks like a problem.

Drew-



More information about the PLUG mailing list