Re: [rdiff-backup-users] Incredibly slow i/o to NAS server

[Robert Nichols]

On 11/28/2016 07:31 PM, Andrea Bolandrina wrote:
> Hi,
>
> I'm running an rdiff-backup script, to backup my laptop to my local NAS.
>
> I love rdiff-backup and it normally works great, but at the moment I'm having 
> problem with one specific directory.
> Such directory is where is store my docker images, therefore it's a lot of 
> files.
> To be precise:
> sudo find /mnt/vms/docker/ -type f|wc -l
> 852443
> sudo du -hs /mnt/vms/docker
> 4.9G    /mnt/vms/docker
> So, it's nearly a million files, but less than 5GB.
>
> When I run the backup the first time, it didn't take too long (can't remember 
> how long exactly, but less than a couple of hours).
> Then I've updated some docker images, and removed the old ones (clearly lots of 
> changes).
> Now I re-run the rdiff-backup and it has been running for nearly 24 hours on 
> that specific folder, no problem with all the other folders.

First, note that the files (except for those with a suffix known to be 
uncompressible) in those removed images is going to be compressed and stored as 
a .snapshot.gz file. That can take quite a while. Even something as simple as 
deleting an old kernel adds 15 minutes or so to my next backup as all the 
deleted files under /usr/src/kernels and /lib/modules get compressed and saved 
away.

Second, are there a large number of files with more than one hard link ("find /mnt/vms/docker/ -type f -links +1 | 
wc -l")? Those can be an issue if the device number (st_dev) changes ("Device:" in the output from the 
"stat" command). When that happens, every one of those multi-link files will be seen as changed and have a 
zero-diff file created in the increments directory. You can avoid that by using the "--no-compare-inode" 
option, but doing that aggravates the problems that rdiff-backup already has with hard-linked files.

--
Bob Nichols     "NOSPAM" is really part of my email address.
                Do NOT delete it.