Re: [Duplicity-talk] Big bandwidth bill from Rackspace

[edgar . soldin]

On 11.06.2013 00:56, Nate Eldredge wrote:
> On Mon, 10 Jun 2013, Joseph D. Wagner wrote:
> 
>> On 06/10/2013 12:20 pm, address@hidden wrote:
>>> yes. essentially it restores any file of the latest backup and compares
>>> it to the original path in the local file system. therefore pretty much
>>> all the last full and the incremental volumes have to be downloaded
>>> again.
>>
>> Why does it need to download the entire file?  Why not just download a list 
>> of files and their checksums (CRC32/64 or if paranoid SHA1/256/512)?  This 
>> would minimize bandwidth usage for verify operations and be just as 
>> effective as comparing the whole file.
> 
> This would tell you whether your local file had changed, but it wouldn't 
> actually verify the backup.  Suppose some of the backup volumes are 
> corrupted, but the signature / checksum files are intact.  Your proposal 
> would not detect this.
> 
> I don't think there's any way to truly verify the backup other than fetching 
> all the volumes, since we don't assume that we can run code or store 
> encryption keys on the backend.
> 

valid point. btw. the next release will have a --compare-data switch which was 
not exposed before to actually compare local path with the backed up version 
bit by bit.

that's the essential difference between a
verification - which actually makes sure everything is fine
vs.
comparison - which might simply look for differences, without restoring anything

duplicity's current verify is a combination of both. we should probably make 
that clearer, or clean it up by separating the actions properly
compare - actually compare local path with a restored copy
verify - simply restore and compare against the saved checksum

but that still would download the volumes. giving compare a parameter to 
compare against the manifest files, saving traffic, would allow to check for 
changes. but that is essentially what --dry-run is doing already. so why bother?

to come back to the original problem, which used 'duply profile 
backup_verify_purge', probably from a cron job. the periodical verification 
would make sense only if you want to make sure the remote backup is not 
corrupted. therefor i'd say Nate's evaluation is correct for this scenario. no 
way around the traffic.

..ede/duply.net