On 2/16/24 08:44, Dominic Raferd wrote:
Until then, I am interested in your parallel processing approach. Presumably
you start 8 parallel rdiff-backup verify sessions for datetime points -1 to -8
(and then, when they are all complete, -9 to -16, -17 to -23...)? And you run 8
in parallel because your CPU has 8 cores?
I have 16 cores, actually, but by experiment I found that 8 parallel threads
seems to be the sweet spot. I don't know how much of that is unique to my
system and the nature of my backups. I did have to add a pre-scan of the
file_statistics metadata files to look for increment sizes of 1GB or greater,
and limit the number of parallel checks to 1 if any are found. All it takes is
one huge ISO file in the increments to gobble up cache and make the parallel
checks really slow. I haven't spent much time trying to tune that adjustment,
and all the experimenting was done back when I had just 32GB of RAM.
I let the parallel threads run independently, without waiting for anything in
the others. Effectively, I run the threads with the level sequences:
{-1..-99..8}
{-2..-99..8}
{-3..-99..8}
...
{-8..-99..8}
and then just wait for everything to complete.
The code is really nothing like that, but that is the effect. You might expect
the threads to get badly out of sync, but because of the effects of I/O
caching, whichever threads finish a step first find themselves slowed down by
I/O waits more than do the threads that advance to a new step later. The
threads tend to stay quite beautifully in sync. Again, that's on my system with
my backups. YMMV.