[Duplicity-talk] silent failure with large backups to cloud files

[Andrew Pimlott]

I back up a large system to cloud files.  Recently, my incremental
backups stopped working: every night, I would see a message like

    Deleting local
    
/var/cache/duplicity/app2.tausdata.org/duplicity-inc.20100829T100001Z.to.20100902T100001Z.manifest
    (not authoritative at backend).

... and it would proceed to create an incremental backup from the last
full backup, not from last night's incremental.

The problem as it turns out:  duplicity tries to list the container
contents in cloud files.  cloud files silently returns only 10000
entries, an API limit.[1]  My backup comprises more than 10000 volumes,
so the recent incremental backups aren't in the listing.  duplicity
thinks they don't exist and starts from the latest manifest that happens
to fall within the 10000 entries.

I think python-cloudfiles is at fault for not even documenting this
limit.  I filed an issue against python-cloudfiles.[2]  There is a way
to get a full listing that could be implemented in duplicity.  You just
request another listing starting where the last one left off.

But I wonder why duplicity is listing the container in the first place.
Listing is an expensive operation, especially in a distributed system.
duplicity should rely on its own manifest instead.

Finally, "Deleting local ... (not authoritative at backend)" is a
cryptic message.  It happened for a few days before I realized what was
going on.  Maybe it could be made a fatal condition, or at least the
warning could be stronger, since it suggests corruption of the backup.

Andrew

[1] https://api.mosso.com/guides/cloudfiles/cf-devguide-20090311.pdf
[2] http://github.com/rackspace/python-cloudfiles/issues/issue/8