Re: using --remove-older-than takes a very long time?

[Eric L.]

Hi Derek,

On 07/02/2020 14:46, Derek Atkins wrote:
> HI,
> 
> I've been running backups on my servers for a long time using
> rdiff-backup.  My backup server has been offline for about a week and I
> just put it back online yesterday.  The server does the following:
> 
> foreach host (list of machines to backup); do
>   rdiff-backup host local-storage-for-host
>   rdiff-backup --force --remove-older-than 1Y local-storage-for-host
> end
> 
> This process starts at 1am.  I noticed today (it's not 8am) that the
> remove process for the first server started at 4:09am and is still
> running.  In other words, the "clear history" process is taking longer
> than the backup did!
> 
> Is this expected behavior?

I don't think so but it probably depends on how many backups it has to
remove.

> 
> While I keep track of how long it takes to run the backup process for
> each server, I didnt separate out the backup vs cleanup stages of the
> process.  If cleanup is going to take so long, I might separate it out
> into a different stage!  Although it would be nice if cleanup could be
> sped up!
> 
> Is there any way to profile the code to see where it is spending its
> time?

A profiling as such is possible at python level but I would start with
just calling the removal option with `-v9`, it should give you/us a
first hint as the date/times for each action are logged.

KR, Eric

> 
> Thanks,
> 
> -derek
> 
> PS: I should mention this is with 1.2.8, not a more recent version.

I wouldn't expect a big difference but you never know...