Re: [rdiff-backup-users] KeyError exception

[Austin Clements]

Nothing obviously unusual.  Lots of normally named directories and a
few symlinks.

$ ls -aF /

./    boot/   etc/     initrd.img@  mnt/   root/  sys/  var/
../   cdrom@  home/    lib/         opt/   sbin/  tmp/  vmlinuz@
bin/  dev/    initrd/  media/       proc/  srv/   usr/

Is there some other --include / --exclude setup I should try?

Quoth Andrew Ferguson on Jan 18 at  1:47 am:
> What are the files in:
> 
> ls -ad /
> 
> ? On my non-working system, there are some /.foo -style directories,
> which get picked up by how you structured the --include / --exclude setup.
> 
> 
> Andrew
> 
> Austin Clements wrote:
> > Intriguing.
> > 
> > It fails on both of my systems, which are both running Python 2.4.4
> > and OpenSSH 4.6p1/OpenSSL 0.9.8g.  I also tried upgrading and running
> > it with OpenSSH 4.7p1/OpenSSL 0.9.8f, and nothing changed.  Then I
> > tried running it through Python 2.5.1 (albeit, with the C modules
> > compiled for 2.4, but that didn't seem to affect anything), and still
> > nothing changed.
> > 
> > I'm not convinced that it's a problem with Python or OpenSSH versions.
> > Can you think of any other significant differences between your
> > working and non-working hosts?
> > 
> > Quoth Andrew Ferguson on Jan 18 at  1:23 am:
> >> Hi Austin,
> >> 
> >> I have now found a system that fails with your script, and have two that
> >> do not. On the one that fails, I am running Python 2.3.4, and on the two
> >> that work, one is running 2.4.1 and the other 2.5.1. The failing system
> >> also has OpenSSH 3.9p1. The working systems have OpenSSH 4.1p1 and 4.5p1
> >> respectively.
> >> 
> >> 
> >> What versions of Python and SSH are you running?
> >> 
> >> 
> >> Andrew
> >> 
> >> Austin Clements wrote:
> >> > The following bug is still present in rdiff-backup 1.1.15.  For
> >> > convenience, I've attached the test script that causes rdiff-backup to
> >> > fail.
> >> > 
> >> > Quoth myself on Sep 21 at  5:50 pm:
> >> >> After a great deal of pain and suffering, I've managed to reduce this
> >> >> problem to an easily reproducible test case.  Put 1999 files or more
> >> >> in a directory and try to back that up, using a remote schema for both
> >> >> the source and the destination.  Below is a script that will do the
> >> >> whole process.
> >> >> 
> >> >> 
> >> >> More specifically, (Globals.pipeline_max_length*4-1) is the cut off.
> >> >> This formula is based on DestinationStruct.set_rorp_cache, which
> >> >> initializes the CacheCollatedPostProcess.  I still really don't
> >> >> understand what's going on here, but it has something to do with the
> >> >> order that the CacheCollatedPostProcess cache is traversed on the
> >> >> destination rdiff-backup server when the source is also an
> >> >> rdiff-backup server.
> >> >> 
> >> >> I generated some simple traces of calls to important methods in the
> >> >> cache and it looks like the order of traversal versus the order in
> >> >> which methods like get_mirror_rorp are called on the destination is
> >> >> radically different depending on whether or not the source uses a
> >> >> remote schema.  I'm not sure, but I think each file in the cache only
> >> >> gets examined once in the correct case (an index is retrieved from the
> >> >> cache, then a bunch of methods are called to get more information
> >> >> about it from the cache, then that index is never looked at again).
> >> >> In the broken case with a source server, I think the cache is iterated
> >> >> through once, and _then_ asked for information about the indexes that
> >> >> were retrieved from it.  Thus, if the first traversal fills up the
> >> >> cache and causes something to be evicted (ie, if there are 1999 files
> >> >> or more), the "second" traversal won't be able to retrieve the
> >> >> additional information and will raise a KeyError.
> >> >> 
> >> >> I'll keep poking at it, but at this point I'm in way, way over my head
> >> >> on the details of how rdiff-backup works, so it's not likely I'll get
> >> >> much further on my own.
> >> >> 
> >> >> 
> >> >> #!/bin/bash
> >> >> 
> >> >> # Cause rdiff-backup to throw a KeyError
> >> >> 
> >> >> set -e
> >> >> 
> >> >> mkdir /tmp/lotsoffiles
> >> >> for (( N=0 ; N < 500*4-1 ; N++ )); do
> >> >>     touch /tmp/lotsoffiles/`printf '%04d\n' $N`
> >> >> done
> >> >> 
> >> >> rdiff-backup \
> >> >>     --terminal-verbosity 5 \
> >> >>     --include /tmp/lotsoffiles \
> >> >>     --exclude / \
> >> >>     localhost::/ localhost::/tmp/lotsoffiles-backup
> >> >> 
> >> >> > 5) Provide your kernel version and details of the source and dest
> >> >> > filesystems (type, any mountpoints involved in the backup).
> >> >> 
> >> >> Source:
> >> >>   Kernel 2.6.18, Mostly Reiser (/) with a small dash of ext3 (/boot)
> >> >> 
> >> >> Destination:
> >> >>   Kernel 2.6.21, Reiser mounted via a crypsetup mapper of a USB drive
> >> >> 
> >> >> I ran my above test script on my source machine (and it didn't include
> >> >> /boot, so it was just Reiser).
> >> > 
> >> > 
> >> > 
> >> > ------------------------------------------------------------------------
> >> > 
> >> > #!/bin/bash
> >> > 
> >> > # Cause rdiff-backup to throw a KeyError
> >> > 
> >> > set -e
> >> > 
> >> > # Create a directory to back up, containing
> >> > # Globals.pipeline_max_length*4-1 (1999) files.  This is the exactly
> >> > # cut off, if you make it 1998, everything works.
> >> > mkdir /tmp/lotsoffiles
> >> > for (( N=0 ; N < 500*4-1 ; N++ )); do
> >> >     touch /tmp/lotsoffiles/`printf '%04d\n' $N`
> >> > done
> >> > 
> >> > # Back up /tmp/lotsoffiles to /tmp/lotsoffiles-backup, using remote
> >> > # access for both.
> >> > rdiff-backup \
> >> >     --terminal-verbosity 5 \
> >> >     --include /tmp/lotsoffiles \
> >> >     --exclude / \
> >> >     localhost::/ localhost::/tmp/lotsoffiles-backup
> > 
> 

-- 
Austin Clements                                          MIT/'06/CSAIL
address@hidden                         http://web.mit.edu/~amdragon/
       Somewhere in the dream we call reality you will find me,
              searching for the reality we call dreams.