[PLUG] Backing up data to DVD

Thu Aug 26 15:11:02 UTC 2004

On Thu, 26 Aug 2004, Roderick A. Anderson raanders-at-acm.org |PDX Linux| wrote:

> Steve, you willing to share your scripts?  I'd like to see how someone
> else would do it.  I know I have my set of hammers.  Be interesting to
> see what other tools are available from the shed.

Gladly!  It's attached and is a direct descendant of the script I posted
here before:

http://lists.pdxlinux.org/pipermail/plug/2003-November/025838.html

Some of its features:
  + GPL licensing
  + uses LVM snapshots to ensure consistent backup
  + backs up to a convenient 3-files-per-DVD size
  + supports incremental backups
  + heavily commented

Some of its problems:
  + dump 0.4b37 recommended if using dump files > 2GB
  + smallest unit backed up is a whole filesystem
  + does not auto-detect what to back up
  + requires configuration changes inside the script before use

  -- Steve
-------------- next part --------------
#!/bin/sh

# Backs up system to a second local drive using "dump".  The only parameter
# is the "dump" incremental level.  This script has only been tested
# on Linux using ext2 dump.

# BEFORE USING THIS YOU MUST:
#  + change the VG_NAME to match the volume group that the LVs you want
#    to back up are in.  If you don't use LVM, use "VG_NAME="
#  + enter the devices you plan to dump without the /dev prefix, e.g.
#    /dev/hda1 -> hda1
#  + put your E-mail address below so the script knows where to send
#    failure messages

VG_NAME=WS3
DEVS_TO_DUMP="hdc1 WS3/WS3_home WS3/WS3_root"
MAIL_TO=root at localhost

# You must make a level 0 backup before automating this script.  It
# will still work, but your first level 1 in essence becomes a level 0.

# Below is a sample schedule of system backups with level 1 on 
# Saturday.  Adjust schedule as makes the most sense for your 
# activity schedule.  The below "Tower of Hanoi" sequence
# is highly recommended since it allows for a maximum number of points
# in time available for recovery with a minimum amount of storage usage.
# (At the expense of some additional complexity during restores.)

# To restore a given day's data:
# DoW     Levels needed
# ------ --------------
# Sat(6): 1
# Sun(0): 1,3
# Mon(1): 1,2
# Tue(2): 1,3,5
# Wed(3): 1,2,4
# Thu(4): 1,3,5,7
# Fri(5): 1,2,4,6

# Sample crontab entries for the above schedule:

#00 06 * * 6 root /root/backup.sh 1 >> /var/tmp/backup.cron.out 2>&1
#00 06 * * 0 root /root/backup.sh 3 >> /var/tmp/backup.cron.out 2>&1
#00 06 * * 1 root /root/backup.sh 2 >> /var/tmp/backup.cron.out 2>&1
#00 06 * * 2 root /root/backup.sh 5 >> /var/tmp/backup.cron.out 2>&1
#00 06 * * 3 root /root/backup.sh 4 >> /var/tmp/backup.cron.out 2>&1
#00 06 * * 4 root /root/backup.sh 7 >> /var/tmp/backup.cron.out 2>&1
#00 06 * * 5 root /root/backup.sh 6 >> /var/tmp/backup.cron.out 2>&1

# The simplest way to restore a file is to use 
# "restore -t -A <archive file>" to list the contents of the dump and
# grep for the file(s) that need to be restored.  Then use
# "restore -i -A <archive file> -f <dump file>" to interactively restore them.
# Just run "ls" and "cd" until you find the file you want to restore,
# choose "add <file>" for each one, then choose "extract".

# Note that this script overwrites each level's previous backup with the
# new one.  It also aborts on the first error so any subsequent 
# filesystems will NOT BE BACKED UP.

# In order to ensure that everything works fine, I STRONGLY recommend
# running a "restore -C -f <file> -M -D <mount point>" after you have
# burned/copied these to their final storage point.  If you want to
# avoid wasted write-once media, then run the restore test both
# before AND after copying the backups.  I've seen about 10% failure
# rate at this step after burning to DVD on my Plextor PX-708UF.
# It seems this writer is a bit cranky with anything except the
# highest-end media.  (Fuji/Taiyo Yuden or Maxell)

# For lots of media discussions, try the forums at:
# http://www.cdfreaks.com

# Good luck with the backups!  Hope you never need 'em.  ;-)

# Steve Bonds
# Email: rn5o6kw02 at sneakemail.com
# Copyright 2003, All rights reserved
# Redistribution permitted under the terms of the GNU General Public License
# version 2.0.  http://www.gnu.org/copyleft/gpl.html

bail() {
	MAIL_NEEDED=yes
	echo `date` "[$$]: Error: $1"
	echo `date` "[$$]: Error: $1" >> $MAIL_BODY
	if [ "x$IS_SNAP" = "x1" ]; then
            lvremove -f /dev/${VG_NAME}/${base_device}_snap && \
	       echo "Snapshot LV /dev/${VG_NAME}/${base_device}_snap removed" >> $MAIL_BODY
            IS_SNAP=
        fi
	MAIL_SUBJECT="Fatal backup error on swbonds2"
	send_mail
	exit 1
}	

send_mail() {
	if [ "${MAIL_NEEDED}x" != "x" -a "${MAIL_TO}x" != "x" ]; then
		cat $MAIL_BODY | $MAIL_CMD "$MAIL_SUBJECT" $MAIL_TO
		cat $MAIL_BODY
	elif [ "${MAIL_NEEDED}x" != "x" ]; then
		echo "Subject: $MAIL_SUBJECT"
		cat $MAIL_BODY
	fi
	rm -f $MAIL_BODY
}

MAIL_CMD="mail -s"
MAIL_BODY=`mktemp backup.XXXXXX`

# If a device starts with this, attempt to create a snapshot before backing
# up.  TODO: allow more than one VG to be backed up
BACKUP_AREA=/backups
LEVEL=$1
: ${LEVEL:?"Please specify backup level (0-9), 0=full"}
shift

# Backups contain every file on the system, so it's real important to be sure
# that only root can read them.
umask 077

# Figure out the free space in the VG available for LV snapshots
if [ "x$VG_NAME" != "x" ]; then
    VG_FREE_PE=`vgdisplay $VG_NAME | grep 'Free  PE' | awk '{print $5}'`

   # This value is pretty arbitrary.  There's no way to ensure with
   # certainty that there will ever be enough space for a snapshot
   # under all circumstances without using space equal to the original
   # LV size.
   if [ $VG_FREE_PE -lt 800 ]; then
       bail "Not enough space in VG for snapshot"
   fi
fi

case $LEVEL in
	[0123456789]) # OK to proceed
	;;
	*) echo "Invalid level, must be 0 through 9"
	exit
esac

for device in $DEVS_TO_DUMP; do
	# Tried cd-sized images via -M -B 665600, but 30 disks for 
	# /home is too much.  Try 2.35GB images via -M -B 2406400
        # since strangely, using 4.4GB (4505600) gave files that
        # were really 318MB in size-- rollover bug?  2.35 broken
        # too.  2.0GB = 2097152 (nope)  2096128 -- OK.
        # Newer version of dump (0.4b37) fixes the 2GB rollover problem.

	# Burn three per disk to keep file size under 2GB (1530000)
	# DVD+R(W) has a lower capacity than DVD-R(W), so the files were
	# just a touch too large to fit on 3 per disk with ISO overhead.
        # New -B: 1529900

	# DO NOT BURN THESE AS-IS TO DVDs!
	# You must first create an ISO/UDF
        # filesystem containing the files.  When burned directly, the
        # images get zero-padded to the end of the next DVD/CD block 
        # size which confuses restore.  For example, using growisofs:
        # growisofs -dvd-compat -Z /dev/dvd -R -J -udf file1 file2 file3
	# BE SURE YOUR MKISOFS CAN HANDLE FILES > 2GB AND TEST ON
	# REWRITEABLE MEDIA FIRST TO AVOID COASTERS.  This was tested
	# with mkisofs 2.01a27.

	# Another option is to skip compression.  Since filesystems use
	# 8192-byte blocks they will always be a multiple of 2048
	# bytes.  However, if for any reason the filesystem is not
	# a multiple of 2048 the CDs/DVDs will not be usable for files
	# that cross the media boundary, and this failure will not be
	# detectable unless a 'restore -C' is run.  I eventually opted
	# for uncompressed backups with an external gzip optional after
	# the dump files were created.

	# When compressed, I had mysterious problems with the decompression
	# failing only on DVD media-- the SAME files on the hard drive
	# worked great.  The problem seemed to be that the buffer for
	# holding the decompressed data was smaller for the DVD filesystem
	# vs. the original ext2 filesystem, and some of our data
	# decompresses nicely.  (But then why does it work great on the
	# same ISO mounted loopback?)  This remains a mystery.

	# In either case, it seems that turning off compression is the
	# best choice for offline media.  This may also simplify
	# recovery in the event portions of the media go bad.  Not that
	# I see that happen all the time... <sarcasm>

	# Enough of my yakkin', on to The Script!

        # Filter out '/' chars from the device name for the purpose
        # of naming the files.  Get the base filename for the purpose
        # of creating an aptly named snapshot LV
        noslash_device=`echo $device | sed 's/\\//_/g'`
        base_device=`echo $device | perl -p -e 's/^[^\\/]+\\///g'`

        # The filenames are used several places, so only figure them
	# out once to avoid typing in piles of curly braces.
	DUMP_FILE=${BACKUP_AREA}/level${LEVEL}/${noslash_device}.dump
	ARCHIVE_FILE=${BACKUP_AREA}/level${LEVEL}/${noslash_device}.archive

	# Remove existing backup at the same level from last week.  If you want
	# archives, then copy 'em out before the next backup of the same level.
	# Note that this likely invalidates ALL the backups of higher level from
	# being properly restored in full.  However, it would still be possible
	# to run partial restores depending on what you needed back, so this
	# script won't delete them.  I.e. your level 7 backup may not be any
	# good if the level 6 that preceeded it gets overwritten.
	rm -f ${DUMP_FILE}* $ARCHIVE_FILE

	# Create LV snapshots to ensure a consistent dump.  This works around
	# one of the longstanding "problems" with dump-- that it can create
	# inconsistent backups.  Never mind that every other tool has the same
	# issues.  ;-)
        if echo $device | grep -E "^$VG_NAME\\/"; then
            IS_SNAP=1
            lvcreate -n ${base_device}_snap --snapshot -l $VG_FREE_PE /dev/${device} || bail "Can't create snapshot LV for $device"
            device=${VG_NAME}/${base_device}_snap
        else
            echo "$device is NOT in $VG_NAME, skipping snapshot creation"
        fi 

	# After all that, we finally get to run the dump command!  Finding
	# the right -B parameter has been quite an adventure.
	dump -${LEVEL} -f $DUMP_FILE -A $ARCHIVE_FILE -q -u -M -B 1529900 /dev/${device} && echo `date` "$device backup succeeded" || bail "Problem backing up $device" 
	if [ "x$IS_SNAP" = "x1" ]; then
            lvremove -f /dev/${VG_NAME}/${base_device}_snap
            IS_SNAP=
        fi

	# OPTIONAL:
	# Compress the uncompressed backups.  This is preferable to using -z in dump
	# since restore has had problems reading compressed backups off of DVD
	# media.  (The uncompress buffer ends up too small for some unknown
	# reason.)  If the gzip fails, don't worry about aborting.
	##gzip -v ${DUMP_FILE}*
done