[PLUG] Out of space on device

[drew wymore]

On Sat, May 9, 2009 at 9:34 AM, wes <plug at the-wes.com> wrote:

> On Sat, May 9, 2009 at 6:59 AM, drew wymore <drew.wymore at gmail.com> wrote:
>
> > On Sat, May 9, 2009 at 6:50 AM, Hal Pomeranz <hal at deer-run.com> wrote:
> >
> > > > I ran into a problem on a server I run last night. df was reporting
> no
> > > space
> > > > left and services obviously stopped working. The weird thing is that
> > when
> > > I
> > > > checked all the directories on the mount point they didn't add up to
> > the
> > > > space allocated to the partition. It's a 37GB mounted as / with /var
> > /bin
> > > > /boot /sbin /usr /dev /media /proc /opt and .sys
> > > >
> > > > I removed some un-needed software on the box to get it back up and
> > > breathing
> > > > again but I still have no clue how it filled up, so I have no way of
> > > knowing
> > > > if it'll happen again. Originally IIRC it was only using about 17GB
> of
> > > space
> > > > as the meat of storage is on /home on a seperate partition.
> > > >
> > > > Hitting me with the clue stick would be appreciated or any ideas on
> > where
> > > to
> > > > look would be great.
> > >
> > > I'm guessing you have an "open but unlinked" file someplace.  In other
> > > words, there's a process that's writing a big data file, but that file
> > > was removed.  Since the file has been removed ("unlinked") from the
> file
> > > system, tools like "du" won't find the space used by this file, which
> is
> > > why your file system totals don't match up.  However, the operating
> > system
> > > is unable to reclaim the space until all processes that have the file
> > > open have been terminated.
> > >
> > > How can you find those processes?  "lsof +L1 | sort -nk7" (as root)
> > > will show you all "files that are open with link count less than 1",
> > > (i.e. the open but unlinked files) and then sort them by size, so you
> can
> > > find the big ones easier.  Use the PID in the second column to kill
> > > the appropriate processes.  Note that not all open but unlinked
> > > files are necessarily a problem.  It's my experience that programs
> > > like VMware and Wine use open but unlinked files a lot.
> > >
> > > --
> > > Hal Pomeranz, Founder/CEO      Deer Run Associates
> hal at deer-run.com
> > >    Network Connectivity and Security, Systems Management, Training
> > > _______________________________________________
> > > PLUG mailing list
> > > PLUG at lists.pdxlinux.org
> > > http://lists.pdxlinux.org/mailman/listinfo/plug
> > >
> >
> > Thanks Hal,
> > I found some processes related to Apache and MySQL which aren't a
> problem.
> > Nothing else showing up that looks like a problem.
> >
> >
> They could become a problem though. If you have a log file filling up a
> partition, and you delete it, it won't go away til you kill the process
> (which is what you just learned).
>
> I have a trick which works around this. If you blank the file, rather than
> deleting it, it will free up the space without killing the process. I use
> echo "" > /var/log/logfile to accomplish this.
>
> -wes
> _______________________________________________
> PLUG mailing list
> PLUG at lists.pdxlinux.org
> http://lists.pdxlinux.org/mailman/listinfo/plug
>


I did find a file in /var/log that was partly to blame, I also removed some
un-needed packages. However I'm still trying to figure out where this space
went. Tim mentioned the mount issue which might be a clue, I had a
mountpoint of /backups pointing to an external USB device which was having
issues (turned out to be an issue with the filesystem, I changed from reiser
to ext3 and the problem went away) ... so the question I have, my
understanding is that if /backups was having issues the parent partition
which is / which is a completely different device, it is possible that junk
was getting written to / because of the issue with /backups ? If that is
indeed the case, I'm wondering where I can find that garbage data which is
sucking up space. I have gone through each of the dir's mounted under / and
they don't add up to the amount showing up in df ... I have all my configs
and required packages that I built from source available to me so starting
over wouldn't take very long. I'm just wondering if there is a way to figure
this out before going to that extreme.

Would the fact that the partition was created with 1k blocks instead of 4k
blocks?

Thanks,
Drew-