Re: [rdiff-backup-users] Safe to mirror backup dir with rsync?

[Dominic Raferd]

Steven Willoughby wrote:
> Use rsync's --delay-updates.  That will narrow the window of failure 
> significantly.
>
> If that's not enough, create a temporary hard-linked copy of /backup on 
> machine 2:
>
> set -e
>
> cp -al /backup /backup.new
> rsync -a --del machine1:/backup/ /backup.new/
> mv /backup /backup.old
> mv /backup.new /backup
> rm -r /backup.old

Thanks Stephen. Your contributions are always very interesting. I 
implemented both of your great tips for updating my secondary backup 
machine (mirror of my rdiff-backup primary): --delay-updates for short 
rsync sessions and the hard-link approach for long ones. (But I think 
there should be no final slash on the rsync line.)

A refinement to the hard-link approach is that the 'cp -al' line is not 
needed to pre-populate /backup.new with hard-links to the files 
referenced by /backup; instead run rsync with --link-dest option - this 
creates the hard-links automatically, which is faster and more elegant. 
I also put rsync inside a loop - so it is rerun if it fails. I don't use 
'set -e' but instead have error-checking built-in.

Result is something like the code below. I use it for a 160GB nightly 
synch (1.4 million files) over a 1MB internet connection - it takes c.4 
hours. [My actual in-use code is more complex than shown below because 
it is a push operation not a pull, it wakes the remote machine at the 
start and puts it back to sleep at the end, it synchs users and groups, 
it synchs from an LVM snapshot which is created for this mirror 
operation and dropped afterwards, and it synchs a whole filesystem (LV 
/home).]

I think the risk of corrupted data on the destination is low - provided 
the source data is whole and consistent (LVM snapshot helps here). An 
rsync failure cannot cause corrupted data, the risk is of the 
destination machine failing while mv'ing the folders after rsync. Since 
this is largely independent of the risk of failure of the source 
machine, I think it is acceptable.

Dominic

#!/bin/bash

# make sure we start with empty /backup.new
# (see --link-dest in man rsync for why)
mkdir -p /backup.new    # create /backup.new if not existing
rm -rf /backup.new/*    # remove any contents from /backup.new

for (($loop=1; $loop<=10; $loop++)); do # until success up to 10x

  rsync -az --link-dest:/backup machine1:/backup/ /backup.new

  if [ "$?" -eq 0 ]; then
    # rsync has completed successfully, now
    # move /backup.new into place as /backup
    mv /backup /backup.old # temporary location for old backup
    if [ "$?" -eq 0 ]; then
      mv /backup.new /backup # move new backup into place
      if [ "$?" -eq 0 ]; then rm -r /backup.old; fi # bin old backup
    fi
    break
  fi
  sleep 60s
  # If looping back after a failed rsync session then any contents
  # already in /backup.new remain, so pickup where we left off.
  # We don't use --partial, it might slow things down [untested].
done

# exit with code 1 if there was a problem
# note that the original destination data should still be found at
# /backup or perhaps at /backup.old
if [ -d /backup.new -o -d /backup.old ]; then exit 1; fi