Re: too many lstat() syscalls, therefore too many IOPS

[griffin tucker]

there is a tool called ionice that comes with the util-linux package,
which should help other read/write processes if you are multitasking.

ionice shouldn't slow down rdiff-backup if there are no other io
processes going on.

On Thu, 13 May 2021 at 03:08, Eric L. Zolf <ewl+rdiffbackup@lavar.de> wrote:
>
> Hi,
>
> first, I don't see anything surprising in what you describe, so all
> normal AFAICJ.
>
> Second, rdiff-backup needs to check each source file/directory and each
> target, compare them and then copy (or not), so if you have some 2300
> files to backup, that would sound about right. If the target or the
> source file doesn't exist, it would give an error.
>
> If the files are small or don't have changes, the lstat happen a lot and
> nothing much else; this is typical random access. It gives a much
> different access pattern than the copying of bigger files, where more
> sequential is typically done to read/write the file's data.
>
> There is no real way to improve the situation, rdiff-backup goes as fast
> as it can and I personally don't know an I/O-equivalent of "nice" (and
> if you limit the I/O, the backup will be even slower).
>
> You could try the --no-fsync option to improve speed:
>
>    --fsync, --no-fsync   [opt] do (or not) often sync the file system
> (_not_ doing it is faster but can be dangerous)
>
> And, yes, the `rdiff-backup-data/increments` directory is used by
> rdiff-backup to keep track of file and directory changes.
>
> Hope this helps,
> Eric
>
> On 12/05/2021 07:10, Andrei Enshin via Any discussion of rdiff-backup wrote:
> >
> > Hi rdiff-backup folks,
> >
> > Since recent, during backing up I can see spike in IOPS up to 500 which 
> > exhaust limit of a VM. Therefore backup process takes very long. I've 
> > straced a bit and what I can see is: many failed  lstat() syscalls:
> > % time     seconds  usecs/call     calls    errors syscall
> > ------ ----------- ----------- --------- --------- ----------------
> >   42.71    0.040247           9      4608      1420 lstat
> >   35.41    0.033370          12      2860           getdents
> >    9.41    0.008865           6      1431           open
> >    4.63    0.004363           3      1430           close
> >    4.03    0.003797           3      1431           fstat
> >    3.75    0.003536           2      1417           getuid
> >    0.04    0.000039          39         1           unlink
> >    0.01    0.000013           1         9           read
> > ------ ----------- ----------- --------- --------- ----------------
> > 100.00    0.094230                 13187      1420 total
> > Seems rdiff-backup checks existence of some file/dir:
> > 10:13:16 lstat("/some/path/rdiff-backup-data/increments/foo/bar", 
> > 0x7ffd832fa810) = -1 ENOENT (No such file or directory) <0.000020>
> > After backup is done, there is still no such file.
> > Seems the part in path -  /rdiff-backup-data/increments/ - is some "config" 
> > for rdiff-backup and probably it tryies to find something but can't?
> >
> > What might be wrong in my setup? What would you recommend to check to solve 
> > the issue if it is issue at all?
> >
> > ---
> > Best Regards,
> > Andrei Enshin
> >
>