[PLUG] Massive buffer cache yet still swapping

[Daggett, Steve]

Steve Bonds:
> I've recently run into a performance problem on my desktop where the
> system swaps out things that I'd rather stay in memory (i.e. 
> the Mozilla

<SNIP>

  Unfortunately, the VM system has been changing a lot recently.  The
/proc/meminfo file hasn't been very well documented since around 2.2.x.

  I was fighting with memory leaks in a kernel module last year and wrote
this little script.  It reads /proc/meminfo and stores the data in a daily
CSV file.  Run the script from cron every X minutes, depending on the
resolution you want to see.  

  Then, It's a simple matter to load the CSV into your spreadsheet of choice
and make charts from the data.  I found the trend data much more useful than
random numbers in a file.  

  The linux kernel will normally use all the memory available.  It will only
savage free cache memory as needed.  The VM shouldn't start swapping until
it runs out of free cache.  The Buffers and Cache are two different memory
pools.  If you're going into depletion because Buffer memory is depleting
real storage... then you have a kernel problem.  Buffer memory is all kernel
space.

  I have some very cool charts I can show you that have that exact same
condition. ;-)  I had a leaky kernel module blowing up memory every few
days.  First comes depletion, then comes swap, then comes thrash, then comes
crash!  The developer and I pulled our hair out for a month finding (and
fixing) all the leaks.  

Steve D...

<BEGIN SCRIPT>
#!/bin/bash

############################################################################
##
dt=`date +%m%d`
memfile="$HOME/mem_"$dt".csv"
tmpfile="$HOME/.mem_temp.txt"
dts=`date +%m/%d/%y,%H:%M:00`          # get the date/time stamp

# copy /proc/meminfo to a working file, strip whitespace
touch $tmpfile
cat /proc/meminfo | sed -e "s/   / /g; s/   / /g; s/  / /g; s/  / /g" >
$tmpfile

############################################################################
##
# Cut the meminfo data into CSV format
# dis the bytes, only record the kbytes
############################################################################
##
# tot_byte=`  grep "Mem:" $tmpfile | cut -d" " -f2`
# used_byte=` grep "Mem:" $tmpfile | cut -d" " -f3`
# free_byte=` grep "Mem:" $tmpfile | cut -d" " -f4`
# share_byte=`grep "Mem:" $tmpfile | cut -d" " -f5`
# buff_byte=` grep "Mem:" $tmpfile | cut -d" " -f6`
# cashe_byte=`grep "Mem:" $tmpfile | cut -d" " -f7`
# swap_tot_byte=` grep "Swap:" $tmpfile | cut -d" " -f2`
# swap_used_byte=`grep "Swap:" $tmpfile | cut -d" " -f3`
# swap_free_byte=`grep "Swap:" $tmpfile | cut -d" " -f4`

tot_Kb=`grep "MemTotal:" $tmpfile | cut -d" " -f2`
free_Kb=`grep "MemFree:" $tmpfile | cut -d" " -f2`
share_byte=`grep "MemShared:" $tmpfile | cut -d" " -f2`

buff_Kb=`grep "Buffers:" $tmpfile | cut -d" " -f2`
cashe_Kb=`grep "^Cached:" $tmpfile | cut -d" " -f2`
swapcashe_Kb=`grep "SwapCached:" $tmpfile | cut -d" " -f2`

active_Kb=`grep "Active:" $tmpfile | cut -d" " -f2`
inactive_Kb=`grep "Inactive:" $tmpfile | cut -d" " -f2`

hi_tot_Kb=`grep "HighTotal:" $tmpfile | cut -d" " -f2`
hi_free_Kb=`grep "HighFree:" $tmpfile | cut -d" " -f2`
lo_tot_Kb=`grep "LowTotal:" $tmpfile | cut -d" " -f2`
lo_free_Kb=`grep "LowFree:" $tmpfile | cut -d" " -f2`
swap_tot_Kb=`grep "SwapTotal:" $tmpfile | cut -d" " -f2`
swap_free_Kb=`grep "SwapFree:" $tmpfile | cut -d" " -f2`

############################################################################
##
# Create the file and write the header line if the file has been (re)moved. 
# dis the bytes, only record the kbytes
############################################################################
##
if [ ! -f $memfile ]; then
#  echo -n
"Date,Time,MemTotByte,MemUsedByte,MemFreeByte,MemShareByte,MemBuffByte," >>
$memfile
#  echo -n
"MemCasheByte,SwapTotByte,SwapUsedByte,SwapFreeByte,MemTotalKb,MemFreeKb,"
>> $memfile
  echo -n "Date,Time,MemTotalKb,MemFreeKb," >> $memfile
  echo -n
"MemSharedKb,BuffersKb,CachedKb,SwapCachedKb,ActiveKb,InactiveKb,HighTotalKb
," >> $memfile
  echo "HighFreeKb,LowTotalKb,LowFreeKb,SwapTotalKb,SwapFreeKb" >> $memfile
fi 

############################################################################
##
# Write out the Memory data in CSV format
# dis the bytes, only record the kbytes
############################################################################
##
# echo -n
$dts","$tot_byte","$used_byte","$free_byte","$share_byte","$buff_byte"," >>
$memfile
# echo -n
$cashe_byte","$swap_tot_byte","$swap_used_byte","$swap_free_byte","$tot_Kb",
"$free_Kb"," >> $memfile
echo -n $dts","$tot_Kb","$free_Kb"," >> $memfile
echo -n
$share_byte","$buff_Kb","$cashe_Kb","$swapcashe_Kb","$active_Kb","$inactive_
Kb","$hi_tot_Kb"," >> $memfile
echo $hi_free_Kb","$lo_tot_Kb","$lo_free_Kb","$swap_tot_Kb","$swap_free_Kb
>> $memfile

############################################################################
##
exit 0
<END SCRIPT>