Re: Why do some folders backup very slowly while all others are fast at the same time?

[EricZolf]

Hi,

before doing any conclusion, I would:

1. backup each folder separately
2. try to backup with another tool (also in parallel), e.g. rsync

And then I would compare the times and only then start to make conclusions.

From all I know, rdiff-backup shouldn't be slower due to bigger amount 
of files (it uses iterators for everything), but many small files are 
generally slower than few big files, so the structure of the data matters.

Personally, I'd rather suspect a congestion of the storage I/O. I'm also 
wondering why the TID of the culprit is so much lower than the others; I 
have no clue whatsoever but it could hint at a difference.

Hope this helps,
Eric

On 21/06/2024 16:52, Eric Robinson wrote:
> Check out the following iotop output. Note that there are 4 rdiff-backups going 
> at the same time, from the same source to the same destination. 3 of the 
> backups are going at 120-200 MB/s, but one of them is only going at 9 MB/s. We 
> have observed this behavior multiple times, and always with the same source 
> folder. The  /zpool0/store51b/db_rsyncs/site816 folder is large compared to the 
> others (around 2TB), but the size of the folder should not impact the transfer 
> speed should it?
>
> Total DISK READ :    1096.30 M/s | Total DISK WRITE :     458.39 M/s
> Actual DISK READ:     157.71 M/s | Actual DISK WRITE:      20.10 M/s
>      TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
> 3289403 be/4 root      128.61 M/s  183.57 M/s  0.00 % 38.48 % python3.6 -s 
> /usr/bin/rdiff-backup --tempdir /zpool0/rdiff_tmpdir --~--force 
> /zpool0/store51b/db_rsyncs/site189 /remote_rdiffs.nfs/site189
> 3232252 be/4 root      163.58 M/s   41.02 M/s  0.00 % 18.51 % python3.6 -s 
> /usr/bin/rdiff-backup --tempdir /zpool0/rdiff_tmpdir --~--force 
> /zpool0/store51b/db_rsyncs/site186 /remote_rdiffs.nfs/site186
> 348290 be/4 root        9.04 M/s    3.67 M/s  0.00 %  1.47 % python3.6 -s 
> /usr/bin/rdiff-backup --tempdir /zpool0/rdiff_tmpdir --~--force 
> /zpool0/store51b/db_rsyncs/site816 /remote_rdiffs.nfs/site816
> 3056376 be/4 root      191.39 M/s  150.88 M/s  0.00 %  0.00 % python3.6 -s 
> /usr/bin/rdiff-backup --tempdir /zpool0/rdiff_tmpdir --~--force 
> /zpool0/store51b/db_rsyncs/site179 /remote_rdiffs.nfs/site179
>
> -Eric
>
>
> Disclaimer : This email and any files transmitted with it are confidential and 
> intended solely for intended recipients. If you are not the named addressee you 
> should not disseminate, distribute, copy or alter this email. Any views or 
> opinions presented in this email are solely those of the author and might not 
> represent those of Physician Select Management. Warning: Although Physician 
> Select Management has taken reasonable precautions to ensure no viruses are 
> present in this email, the company cannot accept responsibility for any loss or 
> damage arising from the use of this email or attachments.