Re: Discussion about file format for the future

[rhkramer]

On Tuesday, June 09, 2020 10:19:39 AM Derek Atkins wrote:
> EricZolf <ewl+rdiffbackup@lavar.de> writes:
> > 3. to answer Derek's e-mail as well: would it have an impact on speed?
> > To be honest, no clue, we would need to analyze this.
> 
> Just as another data point, apparently a year ago my backup server
> wasn't backing stuff up, so for the past few days my nightly backup
> hasn't had anything to do for the "remove incremental > 1 year old" step
> it does.  You know what?  Instead of the process finishing at 6-8pm it's
> finishing before 8am!
> 
> In other words, it takes 7 hours to backup all my systems, and then
> 10-12 hours more to remove all the year-old incrementals.  Yes, the
> removal of snapshots is taking longer than the backups themselves.
> 
> Based on this, if I had unlimited disk space I would never remove an
> incremental.  But disk space is not unlimited, and I figured 1 year was a
> good cutoff.
> 
> I wish this were faster.  My worst offender is one particular system: 3
> hours to backup the server and 8 hours to remove an incremental from that
> dataset.  That seems.... unbalanced.

From the peanut gallery, unbalanced, but not necessarily surprising.  I don't 
know much about rdiff backp, but many things are unsymmetrical with respect to, 
well, let me say, creating vs. uncreating (and it could be in either 
direction) -- I mean consider encryption which depends on the difference in 
speed for factoring a number vs. creating a number (of two primes, iirc).

In the case of rdiff-back, it wouldn't surprise me that diffs (deltas) are 
stored as forward deltas, and, in removing old deltas, a new "base" must be 
created before deleting the deltas.  (My words probably aren't exactly 
correct, I hope they are correct enough to explain my point.)