Re: The Verification Treadmill

[Robert Nichols]

On 2/16/24 08:44, Dominic Raferd wrote:
> Until then, I am interested in your parallel processing approach. Presumably 
> you start 8 parallel rdiff-backup verify sessions for datetime points -1 to -8 
> (and then, when they are all complete, -9 to -16, -17 to -23...)? And you run 8 
> in parallel because your CPU has 8 cores?

I have 16 cores, actually, but by experiment I found that 8 parallel threads 
seems to be the sweet spot. I don't know how much of that is unique to my 
system and the nature of my backups. I did have to add a pre-scan of the 
file_statistics metadata files to look for increment sizes of 1GB or greater, 
and limit the number of parallel checks to 1 if any are found. All it takes is 
one huge ISO file in the increments to gobble up cache and make the parallel 
checks really slow. I haven't spent much time trying to tune that adjustment, 
and all the experimenting was done back when I had just 32GB of RAM.

I let the parallel threads run independently, without waiting for anything in 
the others. Effectively, I run the threads with the level sequences:
     {-1..-99..8}
     {-2..-99..8}
     {-3..-99..8}
     ...
     {-8..-99..8}
and then just wait for everything to complete.

The code is really nothing like that, but that is the effect. You might expect 
the threads to get badly out of sync, but because of the effects of I/O 
caching, whichever threads finish a step first find themselves slowed down by 
I/O waits more than do the threads that advance to a new step later. The 
threads tend to stay quite beautifully in sync. Again, that's on my system with 
my backups. YMMV.

Looking at that code just now, it has become quite hideous over the years. 
Yikes!

--
Bob Nichols     "NOSPAM" is really part of my email address.
                Do NOT delete it.