Re: Re[2]: [rdiff-backup-users] Verify times increasing

[Daniel Miller]

Greg wrote:
> I know Matt corrected this post, but I wanted to address this:
>
> ---
> If you do a --verify-at-time xyz where xyz is your oldest backup, it
> should verify all files in that backup - so every delta should be
> applied. This should verify that all delta's (backups) are good and
> functioning.
>
> [In short, it "verifies" that for each file for which a successful
> verify is returned, that the most current file, all applicable  
> delta's and meta-data
> are good and functioning properly.]

This is what I thought. It is good to have it confirmed. Thanks.

> ---
> However, if files were added after the initial backup, I'd guess that
> a verify won't check the delta's for those files - since they don't
> exist in the set at time xyz
>
> So, while a verify to your oldest backup is good, it's not
> comprehensive for all files that have deltas+meta-data.

This seems to be a weakness in the rdiff-backup verify mechanism. I  
think there is general consensus here on the list that this could be  
improved since what many people are looking for is a way to verify  
that their backup archive (including all past revisions) is free from  
corruption.

> ---
> I'm not aware, so if I'm wrong perhaps someone could correct me, but
> I'd like a command to, in essence, do a comprehensive
> --verify-all-files-in-the-archive. [I'm pretty sure such a thing
> doesn't exist, at least I never saw it in the docs.]
>
> This would apply all deltas to *all* files (back to the oldest copy)  
> and compare the stored
> hashes at the time of backup to the rebuilt file. [Note all the
> files, not just those in a particular target date/delta.]
>
> This wouldn't verify that every file would be correct in every delta
> version, but it would, I think, get as close as one might come to
> that.

I agree, something like this would be great. Although with the speed  
issue's I'm having it may not be practical (i.e. time feasible) to  
reconstruct every file this way before comparing it to a signature  
hash. I would propose that rdiff-backup store some additional meta- 
data which would consist of signature hashes of the delta files as  
they exist on the disk after rdiff-backup is finished with a backup  
(similar to what yafic does - http://www.saddi.com/software/yafic/).  
This should make the verification process much faster (yafic takes  
less than two hours to verify an rdiff-backup repo that takes over  
eight hours to --verify-at-time on my setup). Note that it would not  
replace the --verify-at-time functionality, which would still be  
necessary to verify the integrity of files as they existed before the  
backup. But it would provide a fast way to verify the integrity of an  
rdiff-backup repository.

> Then again, doing an intermediate check of the hash and file at each
> delta point wouldn't take too much longer [or so I think without a
> lot of time invested in pondering it] - so if this option/feature  
> doesn't
> exist and one were to code it, it might not be much more code or
> difficulty...

Ease of implementation may be an argument that favors your proposal.  
My proposal adds a completely new layer of integrity checking on top  
of the existing rdiff-backup functionality.

~ Daniel