[PLUG] Re: [lug] Rsync Backup Solution over SSH

John Sechrest sechrest at peak.org
Tue Oct 21 10:38:02 UTC 2003


Rsync certainly is a useful tool at some level.

We are finding that having a NetApp with hourly snapshots to be a step
above that.

However, the key question to the Rsync process is the reverse process.

suppose that your host is compromised or other wise damaged and
you want to rebuild your system from scratch. What is the path for 
restore from the rsync set up?


Do you have a recipe for that?






"D. Cooper Stevenson" <cstevens at gencom.us> writes:

 % 
 % --=-SCeB/MPAuLSoHA0MoIkO
 % Content-Type: text/plain
 % Content-Transfer-Encoding: 7bit
 % 
 % All,
 % 
 % Here's the scenario: you need a backup solution that will make a
 % complete backup of your server and automatically rotate the backup
 % snapshots using rsync.
 % 
 % These two scripts will take a snapshot once every four hours (depending
 % on your crontab entry) and automatically rotate the snapshots
 % "backward," that is, 'hourly.3' will be deleted, 'hourly.2' will rotate
 % to 'hourly.3,' 'hourly.1' will rotate to 'hourly.2' and so on. 
 % 
 % Moreover, a snapshot of the server will be taken and the three "dailys"
 % will rotate in a similar fashion to the "hourlys" at midnight each day.
 % 
 % Here is an excellent article for doing this:
 % 
 %   http://www-106.ibm.com/developerworks/ibm/library/i-metro17.html
 % 
 %  Why rsync? From the article:
 % 
 % * rsync is fast, stable, and mature; it's been around for years and has
 % served the author well in a multitude of diverse situations.
 % 
 % * rsync was designed to do exactly what we were trying to do -- create
 % an exact replica of a local data repository on a remote system (or
 % visa-versa), and keep the two repositories in "sync" with each other.
 % 
 % * rsync is readily available; it ships with most popular Linux
 % distributions.
 % 
 % While Tom's article is excellent, there's just one catch: this article
 % talks about performing synchronized backups when an NFS mount is
 % available. For security reasons NFS was in my case simply not an option.
 % 
 % I adapted the scripts to work via SSH.
 % 
 % Alert: Automating the scripts via cron requires that root on the origin
 % server be able to log in to the remote server via SSH without a
 % password. Of course, the origin server must still authenticate itself
 % with public/private key authentication. The best way to do this securely
 % is outlined here:
 % 
 %   http://www-106.ibm.com/developerworks/linux/library/l-keyc3/
 % 
 % For your convenience, here is the beginning of the SSH article here:
 % 
 %   http://www-106.ibm.com/developerworks/library/l-keyc.html
 %   
 %  To implement the system, perform the following:
 % 
 % 1) Install the Korne Shell (ksh) if you do not already have it
 % 
 % 2) Copy both hourly_snapshot.sh and daily_snapshot.sh to /usr/bin
 % 
 % 3) Configure the origin server to log in as root via SSH without a
 % password
 % 
 % 4) Create a file called /etc/excludes (example below) and add the
 % directories and/or files you do not wish to back up
 % 
 % 5) Modify the REMOTE_SERVER variable in both scripts to point to the
 % server that will receive the backups and make them executable by root
 % (chmod 744 /usr/bin/script_name.sh)
 % 
 % 6) Modify the SNAPSHOT_RW variable in both scripts to point to the
 % directory on the remote server you wish the system to backup to
 % 
 % 7) Add the following to your crontab: 
 % 
 %   0 */4 * * * /usr/bin/hourly_snapshot.sh
 %   0 0 * * * /usr/bin/daily_snapshot.sh
 % 
 % Since the scripts are so small I have attached them. I hope you will
 % forgive me as sending the information as a small file increases the
 % likelihood of correctness in terms of line breaks, etc. As I mentioned,
 % an example /etc/excludes file is also included.
 % 
 % I wrote the script for the Korne shell. Why? <\start flame here\>
 % because ksh is the Cadillac of all shells. It has air conditioning and
 % power steering. In short, it makes writing scripts really nice.
 % 
 % However, if any of you would like to modify this for Bash, please let me
 % know; I would be very interested to have that.
 % 
 % I hope you find these scripts useful.
 % 
 % 
 % Very Truly Yours,
 % -- 
 % --------------------------------------------------------------
 % | Cooper Stevenson        | Em:  cooper at gencom.us            |
 % | General Computer        | Ph:  541.924.9434                |
 % | "Open For Business"     | Www: http://www.gencom.us        |
 % --------------------------------------------------------------
 % 
 % --=-SCeB/MPAuLSoHA0MoIkO
 % Content-Disposition: attachment; filename=hourly_snapshot.sh
 % Content-Type: text/x-ksh; name=hourly_snapshot.sh; charset=ANSI_X3.4-1968
 % Content-Transfer-Encoding: 7bit
 % 
 % #!/bin/ksh
 % # ---------------------------------------------------------------------
 % # Original script authored by Mike Rubel
 % # Modifications by Tom Syroid, December 17, 2002
 % # Adaptation for SSH by Cooper Stevenson, October 20, 2003
 % # /usr/bin/hourly_snapshot.sh
 % # ---------------------------------------------------------------------
 % #
 % # ------------- system commands used by this script -------------------
 % ID=/usr/bin/id;
 % ECHO=/bin/echo;
 % 
 % RM=/bin/rm;
 % MV=/bin/mv;
 % CP=/bin/cp;
 % TOUCH=/bin/touch;
 % 
 % RSYNC=/usr/bin/rsync;
 % 
 % # ----------- directory/ssh commands used by this script -----------------
 % 
 % SNAPSHOT_RW=/directory/name/of/remote/backup/store;
 % REMOTE_SERVER=hostname_of_your_server;
 % EXCLUDES=/etc/excludes;
 % LS_COMMAND="$(ssh root@"$REMOTE_SERVER" ls -l $SNAPSHOT_RW)";
 % 
 % 
 % # ------------- the script itself -------------------------------------
 % 
 % # make sure we're running as root
 % if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root.  Exiting..."; exit; } fi
 % 
 % # rotating snapshots of /
 % 
 % # step 1: delete the oldest snapshot, if it exists:
 % if [[ "$(print "$LS_COMMAND" | grep hourly.3 | awk '{ print $9 }')" == hourly.3 ]]
 % then
 %     $(ssh root@"$REMOTE_SERVER" $RM -rf "$SNAPSHOT_RW"/hourly.3)
 % fi;
 % 
 % # step 2: shift the middle snapshots(s) back by one, if they exist
 % if [[ "$(print "$LS_COMMAND" | grep hourly.2 | awk '{ print $9 }')" == hourly.2 ]]
 % then
 %   $(ssh root@"$REMOTE_SERVER" $MV "$SNAPSHOT_RW"/hourly.2 "$SNAPSHOT_RW"/hourly.3)
 %   fi;
 % 
 % if [[ "$(print "$LS_COMMAND" | grep hourly.1 | awk '{ print $9 }')" == hourly.1 ]]
 % then
 %   $(ssh root@"$REMOTE_SERVER" $MV "$SNAPSHOT_RW"/hourly.1 "$SNAPSHOT_RW"/hourly.2)
 %   fi;
 % 
 % # step 3: make a hard-link-only (except for dirs) copy of the 
 % # latest snapshot, if that exists
 % if [[ "$(print "$LS_COMMAND" | grep hourly.0 | awk '{ print $9 }')" == hourly.0 ]]
 % then
 %     $(ssh root@"$REMOTE_SERVER" $CP -al "$SNAPSHOT_RW"/hourly.0 "$SNAPSHOT_RW"/hourly.1)
 % fi;
 % 
 % # step 4: rsync from the system into the latest snapshot (notice that
 % # rsync behaves like cp --remove-destination by default, so the destination
 % # is unlinked first.  If it were not so, this would copy over the other
 % # snapshot(s) too!
 % 
 % $RSYNC -avze ssh --delete --delete-excluded --exclude-from="$EXCLUDES" / root@"$REMOTE_SERVER":$SNAPSHOT_RW/hourly.0 ; 
 % 
 % # step 5: update the mtime of hourly.0 to reflect the snapshot time
 % $(ssh root@"$REMOTE_SERVER" $TOUCH $SNAPSHOT_RW/hourly.0) ;
 % 
 % # script ends
 % 
 % 
 % --=-SCeB/MPAuLSoHA0MoIkO
 % Content-Disposition: attachment; filename=daily_snapshot.sh
 % Content-Type: text/x-ksh; name=daily_snapshot.sh; charset=ANSI_X3.4-1968
 % Content-Transfer-Encoding: 7bit
 % 
 % #!/bin/ksh
 % # ---------------------------------------------------------------------
 % # Original script authored by Mike Rubel
 % # Modifications by Tom Syroid, December 17, 2002
 % # Adaptation for SSH by Cooper Stevenson, October 20, 2003
 % # /usr/bin/daily_snapshot.sh
 % # ---------------------------------------------------------------------
 % # intended to be run daily as a cron job when hourly.3 contains the
 % # midnight (or whenever you want) snapshot; say, 13:00 for 4-hour      
 % # snapshots.
 % #
 % # ------------- system commands used by this script -------------------
 % 
 % ID=/usr/bin/id;
 % ECHO=/bin/echo;
 % 
 % RM=/bin/rm;
 % MV=/bin/mv;
 % CP=/bin/cp;
 % TOUCH=/bin/touch;
 % RSYNC=/usr/bin/rsync;
 % 
 % # ----------- directory/ssh commands used by this script -----------------
 % SNAPSHOT_RW=/directory/name/of/remote/backup/store;
 % REMOTE_SERVER=hostname_of_your_server;
 % LS_COMMAND=$(ssh root@"$REMOTE_SERVER" ls -l $SNAPSHOT_RW);
 % 
 % # ------------- file locations ----------------------------------------
 % 
 % EXCLUDES=/etc/excludes;
 % 
 % # ------------- the script itself -------------------------------------
 % 
 % # make sure we're running as root
 % if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root.  
 % Exiting..."; exit; } fi
 % 
 % # step 1: delete the oldest snapshot, if it exists:
 % 
 % if [[ "$(print "$LS_COMMAND" | grep daily.2 | awk '{ print $9 }')" == daily.2 ]]
 % then
 %   $(ssh root@"$REMOTE_SERVER" $RM -rf "$SNAPSHOT_RW"/daily.2)
 % fi;
 % 
 % # step 2: shift the middle snapshots(s) back by one, if they exist
 % if [[ "$(print "$LS_COMMAND" | grep daily.1 | awk '{ print $9 }')" == daily.1 ]]
 % then
 %  $(ssh root@"$REMOTE_SERVER" $MV "$SNAPSHOT_RW"/daily.1 "$SNAPSHOT_RW"/daily.2)
 % fi;
 % 
 % if [[ "$(print "$LS_COMMAND" | grep daily.0 | awk '{ print $9 }')" == daily.0 ]]
 % then
 %  $(ssh root@"$REMOTE_SERVER" $MV "$SNAPSHOT_RW"/daily.0 "$SNAPSHOT_RW"/daily.1)
 % fi;
 % 
 % # step 3: make a hard-link-only (except for dirs) copy of
 % # hourly.3, assuming that exists, into daily.0
 % if [[ "$(print "$LS_COMMAND" | grep hourly.3 | awk '{ print $9 }')" == hourly.3 ]]
 % then
 %   $(ssh root@"$REMOTE_SERVER" $CP -al "$SNAPSHOT_RW"/hourly.3 "$SNAPSHOT_RW"/daily.0)
 % 
 % fi;
 % 
 % # note: do *not* update the mtime of daily.0; it will reflect
 % # when hourly.3 was made, which should be correct.
 % 
 % # script ends
 % 
 % 
 % --=-SCeB/MPAuLSoHA0MoIkO
 % Content-Disposition: attachment; filename=excludes
 % Content-Type: text/plain; name=excludes; charset=ANSI_X3.4-1968
 % Content-Transfer-Encoding: 7bit
 % 
 % # /etc/excludes
 % # This file tells rsync wich filesystems to exclude
 % # to invoke, add --exclude-from=/etc/excludes in the argument to 'rsync'
 % 
 % tmp/
 % mnt/
 % proc/
 % /dev/shm/
 % 
 % --=-SCeB/MPAuLSoHA0MoIkO--

-----
John Sechrest          .         Helping people use
CTO PEAK -              .           computers and the Internet
Public Electronic         .            more effectively
Access to Knowledge,Inc       .                      
1600 SW Western, Suite 180       .            Internet: sechrest at peak.org
Corvallis Oregon 97333               .                  (541) 754-7325
                                            . http://www.peak.org/~sechrest




More information about the PLUG mailing list