[PLUG] Out of space on device

[wes]

On Sat, May 9, 2009 at 6:59 AM, drew wymore <drew.wymore at gmail.com> wrote:

> On Sat, May 9, 2009 at 6:50 AM, Hal Pomeranz <hal at deer-run.com> wrote:
>
> > > I ran into a problem on a server I run last night. df was reporting no
> > space
> > > left and services obviously stopped working. The weird thing is that
> when
> > I
> > > checked all the directories on the mount point they didn't add up to
> the
> > > space allocated to the partition. It's a 37GB mounted as / with /var
> /bin
> > > /boot /sbin /usr /dev /media /proc /opt and .sys
> > >
> > > I removed some un-needed software on the box to get it back up and
> > breathing
> > > again but I still have no clue how it filled up, so I have no way of
> > knowing
> > > if it'll happen again. Originally IIRC it was only using about 17GB of
> > space
> > > as the meat of storage is on /home on a seperate partition.
> > >
> > > Hitting me with the clue stick would be appreciated or any ideas on
> where
> > to
> > > look would be great.
> >
> > I'm guessing you have an "open but unlinked" file someplace.  In other
> > words, there's a process that's writing a big data file, but that file
> > was removed.  Since the file has been removed ("unlinked") from the file
> > system, tools like "du" won't find the space used by this file, which is
> > why your file system totals don't match up.  However, the operating
> system
> > is unable to reclaim the space until all processes that have the file
> > open have been terminated.
> >
> > How can you find those processes?  "lsof +L1 | sort -nk7" (as root)
> > will show you all "files that are open with link count less than 1",
> > (i.e. the open but unlinked files) and then sort them by size, so you can
> > find the big ones easier.  Use the PID in the second column to kill
> > the appropriate processes.  Note that not all open but unlinked
> > files are necessarily a problem.  It's my experience that programs
> > like VMware and Wine use open but unlinked files a lot.
> >
> > --
> > Hal Pomeranz, Founder/CEO      Deer Run Associates      hal at deer-run.com
> >    Network Connectivity and Security, Systems Management, Training
> > _______________________________________________
> > PLUG mailing list
> > PLUG at lists.pdxlinux.org
> > http://lists.pdxlinux.org/mailman/listinfo/plug
> >
>
> Thanks Hal,
> I found some processes related to Apache and MySQL which aren't a problem.
> Nothing else showing up that looks like a problem.
>
>
They could become a problem though. If you have a log file filling up a
partition, and you delete it, it won't go away til you kill the process
(which is what you just learned).

I have a trick which works around this. If you blank the file, rather than
deleting it, it will free up the space without killing the process. I use
echo "" > /var/log/logfile to accomplish this.

-wes