The Verification Treadmill

[Dominic Raferd]

I wondered if those who know the rdiff-backup code from the inside can 
confirm or correct my understanding about verification of rdiff-backup 
repositories, which is as follows:

'rdiff-backup verify' verifies the integrity of all files/directories 
(etc) in a single backup session at the specified datetime. Implicitly 
it verifies the integrity of later versions of any files that existed at 
that datetime. (This is because rdiff-backup uses reverse diffs and when 
recovering from, or verifying a file at, the given datetime it must 
build it by taking the latest version that it holds and then applying 
reverse diffs sequentially to regress the file to the form it had at 
that datetime.)

But this verification at a given datetime does not verify any file that 
was created later than that datetime. Such a file, if subsequently 
deleted from the original source, could prove irrecoverable from the 
repository even though the earlier session had been verified - if there 
had been corruption affecting this file (but not the files that existed 
at the earlier datetime) in more recent session(s).

So the only way to be confident about *all* the data in a repository is 
to use 'rdiff-backup verify' to verify each and every backup session in 
each repository; and this includes verifying the current 'mirror' 
session (even though it is held in the clear in the repository). This 
needs to be done with reasonable frequency to ensure that backed-up data 
has not deteriorated (e.g. through media bitrot).

All of which takes a lot of computing power and time, much of which is 
duplication of effort (because, as stated above, the verification of the 
earliest session in a repository confirms the integrity of all later 
versions of files that it contains, but it is not possible to exclude 
these files from re-verification for more recent sessions).