Re: Problems with backup after using rdiff-backup-delete.py

[Joe Steele]

On 11/4/2019 2:22 PM, David Bristow wrote:
> Hello,
> I had to delete all references to a folder in one of my backups, using
> rdiff-backup-delete.py, which one of the devs basically wrote for us.
> The script completely removes certain files from the backup going back
> to the first time it was used.  This is due to the customer requiring
> that all copies of certain files are removed.
>
> We used rdiff-backup-delete.py to delete a folder, "ibest", last week.
> The folder was removed, but every back up since then has been giving
> errors (see below this message).

Interesting.  I had not heard of rdiff-backup-delete.py before.  It 
apparently was written by Sol1 a couple years ago and was added to the 
github repo:

https://github.com/rdiff-backup/rdiff-backup/commit/858fb21806187aac09ba31286eb4d128f4613410

The utility would appear to be incomplete.  It makes no attempt to scrub 
the extended_attributes.*.gz files and the access_control_lists.*.gz 
files for references to files that are to be deleted.

Your error log implies that rdiff-backup was attempting to perform a 
regression (regress.py), and in the process of doing that, it was 
attempting to deal with extended attributes (eas_acls.py: join_ea_iter) 
when suddenly it could not find a file it needed to process 
(ibest/original_data/gates_study/0506/basicskillssesa56.dta).  I'm 
guessing that that file name came from the extended_attributes file 
which the utility didn't scrub.

Another area where the utility is deficient is its failure to look for 
files that may have been stored under rdiff-backup-data/long_filename_data.

The utility needs some work.

--Joe


> "ibest" was the name of the top-level folder whose contents, including
> all sub-folders, totally about 250 files and about 2.4G storage, had
> been deleted from the source. All metadata and references to these
> folders should have been deleted by rdiff-bacup-delete.py.
>
>
> We are running rdiff-backup-1.2.8-7.el6.x86_64 running on a RHEL box.
>
> Thanks,
> David
>
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 304, in error_check_Main
>      try: Main(arglist)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 324, in Main
>      take_action(rps)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 280, in take_action
>      elif action == "backup": Backup(rps[0], rps[1])
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 337, in Backup
>      backup_final_init(rpout)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 501, in backup_final_init
>      checkdest_if_necessary(rpout)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 920, in checkdest_if_necessary
>      dest_rp.conn.regress.Regress(dest_rp)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/regress.py",
> line 71, in Regress
>      for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index, rf)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/regress.py",
> line 197, in iterate_meta_rfs
>      for raw_rf, metadata_rorp in collated:
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/rorpiter.py",
> line 100, in Collate2Iters
>      try: relem2 = riter2.next()
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/eas_acls.py",
> line 210, in join_ea_iter
>      assert rorp, "Missing rorp for index %s" % (ea.index,)
> Traceback (most recent call last):
>    File "/usr/bin/rdiff-backup", line 30, in <module>
>      rdiff_backup.Main.error_check_Main(sys.argv[1:])
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 304, in error_check_Main
>      try: Main(arglist)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 324, in Main
>      take_action(rps)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 280, in take_action
>      elif action == "backup": Backup(rps[0], rps[1])
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 337, in Backup
>      backup_final_init(rpout)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 501, in backup_final_init
>      checkdest_if_necessary(rpout)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/Main.py", line
> 920, in checkdest_if_necessary
>      dest_rp.conn.regress.Regress(dest_rp)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/regress.py",
> line 71, in Regress
>      for rf in iterate_meta_rfs(mirror_rp, inc_rpath): ITR(rf.index, rf)
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/regress.py",
> line 197, in iterate_meta_rfs
>      for raw_rf, metadata_rorp in collated:
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/rorpiter.py",
> line 100, in Collate2Iters
>      try: relem2 = riter2.next()
>    File "/usr/lib64/python2.6/site-packages/rdiff_backup/eas_acls.py",
> line 210, in join_ea_iter
>      assert rorp, "Missing rorp for index %s" % (ea.index,)
> AssertionError: Missing rorp for index ('ibest', 'original_data',
> 'gates_study', '0506', 'basicskillssesa56.dta')