[rdiff-backup-users] Speed Up full restore By auto generating --exclude-globbing-filelist ???

[john]

Thanks in advance for any help you can provide.

Background:
I lost a raid5 array on the main server which housed a samba share for 
my users....I had a remote ssh server just for holding a nightly mirror 
of the shares with 50 days of diffs of these directories.  I mapped the 
samba shares to read only serve from the remote rdiff-mirror while I did 
a full restore from this most recent mirror to the main server.  If it 
is ever finished, I will switch the samba share back to read write on 
the main server and continue with my nightly rdiff-backups of the 
lvm-snapshot.  Having the backup was a lifesaver.  read-only  with only 
a few hours missed backup and no downtime is nice but its taking too 
long to restore.


Restoring individual files was easy but a full restore is slow 
especially since it keeps stopping.
*It has taken a week so far... mainly because I have to go through a 
simple 3 step process over and over.
*
1. Start the restore
2. Wait until it produces the error which is always related to a 
particular file
3. Copy the file that caused the error to the --exclude-globbing-filelist
4. return to step 1.

The time between errors various, but can be the next file or a hundred 
files later.

I am happy to have the globbing list to try to fix these files later and 
know what is missing...but getting 95% of the files back fast so I can 
get back to read write soon is important.
I want to return the fileserve to read/write asap.

Anyone know a work around that might help?

Thanks,
John















I couldn't find the log, but this is what appears on the ssh 
terminal...  I go from a reiserfs to an ext3 filesystem with different 
usernames on the backup server.

XXXXfileserve.XXX:/mnt/databases/database# rdiff-backup  --restore-as-of 
1D --force --exclude-globbing-filelist 
/mnt/databases/database/exclude-globbing.list 
"address@hidden::/backup/XXXXfileserve.XXX/mnt/snap-home/common/XXXXX" 
"/home/common/XXXXX"
Fatal Error: Error opening file 
/mnt/databases/database/exclude-globbing.list
XXXXfileserve.XXX:/mnt/databases/database# rdiff-backup  --restore-as-of 
1D --force --exclude-globbing-filelist 
/home/database/exclude-globbing.list 
"address@hidden::/backup/XXXXfileserve.XXX/mnt/snap-home/common/XXXXX" 
"/home/common/XXXXX"
-----------------------------------------------------------------
Detected abilities for destination (read/write) file system:
 Characters needing quoting                   ''
 Ownership changing                           On
 Hard linking                                 On
 fsync() directories                          On
 Directory inc permissions                    On
 Access control lists                         Off
 Extended attributes                          Off
 Mac OS X style resource forks                Off
 Mac OS X Finder information                  Off
-----------------------------------------------------------------
-----------------------------------------------------------------
Detected abilities for source (read only) file system:
 Access control lists                         Off
 Extended attributes                          Off
 Mac OS X style resource forks                Off
 Mac OS X Finder information                  Off
-----------------------------------------------------------------
Traceback (most recent call last):
 File "/usr/bin/rdiff-backup", line 23, in ?
   rdiff_backup.Main.Main(sys.argv[1:])
 File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 
260, in Main
   take_action(rps)
 File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 
240, in take_action
   elif action == "restore-as-of": Restore(rps[0], rps[1], 1)
 File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 
465, in Restore
   inc_rpath, dest_rp, time)
 File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 
45, in Restore
   TargetS.patch(target, diff_iter)
 File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 
309, in patch
   ITR(diff.index, diff)
 File "/usr/lib/python2.3/site-packages/rdiff_backup/rorpiter.py", line 
279, in __call__
   last_branch.fast_process(*args)
 File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 
625, in fast_process
   self.patch_to_temp(rp, diff_rorp, tf)
 File "/usr/lib/python2.3/site-packages/rdiff_backup/restore.py", line 
633, in patch_to_temp
   rpath.copy(diff_rorp, new)
 File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 
96, in copy
   if rpin.isreg(): copy_reg_file(rpin, rpout, compress)
 File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 
118, in copy_reg_file
   rpout.write_from_fileobj(rpin.open("rb"), compress = compress)
 File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 
949, in write_from_fileobj
   copyfileobj(fp, outfp)
 File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 
58, in copyfileobj
   inbuf = inputfp.read(blocksize)
 File "/usr/lib/python2.3/site-packages/rdiff_backup/rpath.py", line 
1157, in read
   def read(self, length = -1): return self.file.read(length)
 File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 
117, in read
   if not self.addtobuffer(): break
 File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 
132, in addtobuffer
   type, data = self.iwf._get()
 File "/usr/lib/python2.3/site-packages/rdiff_backup/iterfile.py", line 
401, in _get
   if not self.buf: self.buf += self.file.read()
 File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", 
line 513, in read
   return self.connection.VirtualFile.readfromid(self.id, length)
 File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", 
line 445, in __call__
   return apply(self.connection.reval, (self.name,) + args)
 File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", 
line 367, in reval
   if isinstance(result, Exception): raise result
AssertionError: (('mnt', 'snap-home', 'common', 'XXXX'), ('mnt', 
'snap-home', 'common', 'XXXX', 'XXXX', 'XXXXXXX 12-27-06.doc'))
XXXXfileserve.XXX:/mnt/databases/database# Traceback (most recent call 
last):
 File "/usr/bin/rdiff-backup", line 23, in ?
   rdiff_backup.Main.Main(sys.argv[1:])
 File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 
260, in Main
   take_action(rps)
 File "/usr/lib/python2.3/site-packages/rdiff_backup/Main.py", line 
228, in take_action
   connection.PipeConnection(sys.stdin, sys.stdout).Server()
 File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", 
line 352, in Server
   self.get_response(-1)
 File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", 
line 314, in get_response
   try: req_num, object = self._get()
 File "/usr/lib/python2.3/site-packages/rdiff_backup/connection.py", 
line 230, in _get
   raise ConnectionReadError("Truncated header string (problem "