On 2/16/24 08:44, Dominic Raferd wrote:
Until then, I am interested in your parallel processing approach.
Presumably you start 8 parallel rdiff-backup verify sessions for
datetime points -1 to -8 (and then, when they are all complete, -9 to
-16, -17 to -23...)? And you run 8 in parallel because your CPU has 8
cores?
I have 16 cores, actually, but by experiment I found that 8 parallel
threads seems to be the sweet spot. I don't know how much of that is
unique to my system and the nature of my backups. I did have to add a
pre-scan of the file_statistics metadata files to look for increment
sizes of 1GB or greater, and limit the number of parallel checks to 1
if any are found. All it takes is one huge ISO file in the increments
to gobble up cache and make the parallel checks really slow. I haven't
spent much time trying to tune that adjustment, and all the
experimenting was done back when I had just 32GB of RAM.
I let the parallel threads run independently, without waiting for
anything in the others. Effectively, I run the threads with the level
sequences:
{-1..-99..8}
{-2..-99..8}
{-3..-99..8}
...
{-8..-99..8}
and then just wait for everything to complete.
The code is really nothing like that, but that is the effect. You
might expect the threads to get badly out of sync, but because of the
effects of I/O caching, whichever threads finish a step first find
themselves slowed down by I/O waits more than do the threads that
advance to a new step later. The threads tend to stay quite
beautifully in sync. Again, that's on my system with my backups. YMMV.