Re: using --remove-older-than takes a very long time?

[Derek Atkins]

HI,

On Sat, February 22, 2020 5:37 am, EricZolf wrote:
> Hi Derek,
>
> sorry for not reacting earlier, just overburdened with other things...
> Also absolutely sorry for my first answer, it wasn't going in the right
> direction.

No worries...  Should I turn off -v9?

> On 21/02/2020 18:21, Derek Atkins wrote:
[snip]
> Given that nothing fails but is "only" slow (and I realize that "only
> slow" is also an issue), I hesitate to spend too much time on it for
> now. I would prefer to spend the time on overall performance
> improvements (if any is possible) once we've simplified the code.

I understand.  I suspect that I could use rdiff-backup 1.2.8 to talk to
the host and then rdiff-backup-2/3 to do the removals, right?  That should
work?

>> I'll note that I have not completely ruled out filesystem slowness on
>> deletions.  The underlying file system is EncFS/Fuse over NFS over ZFS,
>> but raw Filesystem I/O tests show good throughput in general.
>
> I wouldn't expect that throughput is the main issue but latency, as it's
> more a question of many files being removed than few big files being
> transferred. Also the raw value isn't really interesting (assuming most
> of the operations done by rdiff-backup happen in cache).

Yes, you're right..  Throughput and latency are both potential issues.  I
ran a bonnie++ experiment last night (which ended 4 seconds into the
rdiff-backup start) to test out the system.  The output is below.

> With the new version of rdiff-backup, you could try the `--no-fsync`
> option, at the slight risk of losing data, if something goes wrong at
> the wrong time.
>
> Check those times on a normal HDD:
>
> $ time for i in $(seq 1 100); do touch $i; sync; done
>
> real  0m7.212s
> user  0m0.191s
> sys   0m0.435s
>
> $ time for i in $(seq 1 100); do rm $i; sync; done
> real  0m7.789s
> user  0m0.209s
> sys   0m0.424s
>
> $ time for i in $(seq 1 100); do touch $i; done
> real  0m0.091s
> user  0m0.044s
> sys   0m0.054s
>
> $ time for i in $(seq 1 100); do rm $i; done
> real  0m0.097s
> user  0m0.042s
> sys   0m0.062s

I can test this out tonight once today's backups are complete.

> We talk about a factor of 70 to 80 between synced and non-synced, and
> it's almost only latency we're talking about here, which I'd expect to
> be even worse in your case, with user space and network involved.
>
> Perhaps your setup would be more performant and with less risk if you'd
> use the remote features of rdiff-backup instead of using NFS, i.e. have
> rdiff-backup being installed and reachable over SSH where you have your
> NFS server running (if at all possible, and without promise that it
> improves performance).

Here is how I run:

[FreeNAS] -- NFS -- [Backup Server] -- SSH -- [Server being Backed up]

FreeNAS uses ZFS as the underlying storage and the Backup server mounts
that as an NFS share.  Then on the backup server it uses EncFS so what
gets stored on FreeNAS is encrypted.  Then it uses rdiff-backup to connect
to the servers being backed up and "pulls" the backup from the target
server.  Finally, after the backup is done, the backup server removes any
increments older than 1 year.

> Hope this helps, because I don't think the issue is solely in the code
> but rather in your setup. We can look at improving performance but I
> don't expect any quick win (beside the slightly dangerous --no-fsync
> option).

I certainly have not ruled this out.  Older tests I did a while ago showed
good performance, but possibly not the "rm'ing thousands of files" use
case.

Thanks,

-derek

> KR, Eric

The Bonnie++ output:

date; time bonnie++ -r 8192 ; date
Fri 21 Feb 2020 09:06:21 PM EST
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.98       ------Sequential Output------ --Sequential Input-
--Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP 
/sec %CP
backup          16G    0k  18 3835k  17 3283k  15  273k  85 54.8m  11
482.3  33
Latency             12246ms     859ms    1633ms    1577ms   10015us     403ms
Version  1.98       ------Sequential Create------ --------Random
Create--------
backup              -Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP 
/sec %CP
                 16     0   8     0   9     0   9     0   8     0   9    
0   8
Latency             10227us    2896us   96684us   11198us    3271us   10385us
1.98,1.98,backup,1,1582413653,16G,,8192,5,0,18,3835,17,3283,15,273,85,56122,11,482.3,33,16,,,,,154,8,685,9,398,9,161,8,724,9,417,8,12246ms,859ms,1633ms,1577ms,10015us,403ms,10227us,2896us,96684us,11198us,3271us,10385us

real    233m43.383s
user    2m50.934s
sys     36m6.502s
Sat 22 Feb 2020 01:00:04 AM EST


-- 
       Derek Atkins                 617-623-3745
       address@hidden             www.ihtfp.com
       Computer and Internet Security Consultant