Re: cross-platform backup tool Unexpectedly Slow on Initial Backup

[Dominic Raferd]

On 08/02/2022 06:59, ewl+rdiffbackup@lavar.de wrote:
> Hi,
>
> just because I became curious, the numbers are probably not to be 
> compared:
>
> - iperf3 tells me ~170Mbit/s
> - I transferred initially my Downloads repository from laptop to 
> server (which has roughly as much disk as you have RAM :-P), 2.7GBytes 
> in 3.45minutes, a mixture of not too big files and quite some small 
> files (see the size distribution below).
> - this makes it roughly 90-100Mbit/s
>
> As said, nothing comparable, just for comparaison. Even on a local 
> backup, the initial backup takes quite a while and becomes better in 
> delta mode.
>
> I have the plan to improve performance, without knowing if there is 
> really potential, but at this stage, I'm focussing on re-structuring 
> the code and understanding it better.
>
> KR, Eric
>
> Neat one liner I found at 
> https://superuser.com/questions/565443/generate-distribution-of-file-sizes-from-the-command-prompt
> (added as utility at 
> https://github.com/rdiff-backup/rdiff-backup/pull/668)
>
> $ find Downloads -type f -print0 | xargs -0 ls -l | awk 
> '{size[int(log($5)/log(2))]++}END{for (i in size) printf("%10d %3d\n", 
> 2^i, size[i])}' | sort -n
>          0   3
>          4   4
>          8   3
>         16   6
>         32   5
>        128   4
>        256   8
>        512  14
>       1024  27
>       2048  29
>       4096  30
>       8192  18
>      16384  15
>      32768   9
>      65536  16
>     131072  15
>     262144  12
>     524288   6
>    1048576   1
>    2097152   3
>    4194304   7
>    8388608   5
>   16777216   2
>   33554432   4
>   67108864   2
>  268435456   1
>  536870912   1
> 1073741824   1

Love the one-liner, here is my variation on it (analysing all the 
current data in my repos, excluding some extraneous stuff and the actual 
reverse-diff data (i.e. the rdiff-backup-data subdirectories) - sorry no 
data re speed:

find /home -type f \( \( -path "/home/lost+found/*" -or -path 
"/home/timedicer/*" -or -path "/home/backup/*" -or -path /home/swapfile 
-or -path "/home/*/.*" -or -path "*/rdiff-backup-data/*" \) -prune -or 
-printf "%s\n" \)|awk '{size[int(log($1)/log(1024))]++}END{for (i in 
size) print 1024^i, size[i]}' | sort -n | numfmt --to=iec 
--format='%.0fB'| sed '2,${s/ /+ /};1iSize Count' | column -t
Size  Count
0B    24284
1B+   88907
1KB+  494536
1MB+  64411
1GB+  59