Re: [rdiff-backup-users] rdiff-backup performance -- slow operation on initial backup?

[Derek Atkins]

Hi,

You are right that it doesn't seem like rdiff-backup alone is the issue;
but to me it sounds like part of the issue is librsync.

>From my original email, I'm getting a good 5 MB/s using raw scp.  But I'm
only seeing 1-2MB/s using rsync and/or rdiff-backup.  So there is clearly
overhead with these two protocols over raw ssh/scp.

Of course scp is still being a bottleneck (compared to 20MB/s capability
on the back-end).  But even if I could get rsync/rdiff-backup up to the
scp-capability of 5MB/s that would help.

I'll take a look at hpn-ssh, but building that for all targets might be a
major PITA.  I might also look at alternate ciphers.

Thanks,

-derek

On Wed, March 30, 2016 3:29 am, Dominic Raferd wrote:
> It doesn't sound like rdiff-backup is the culprit here. You could try
> hpn-ssh https://sourceforge.net/projects/hpnssh/ ?
>
> On 29 March 2016 at 21:03, Derek Atkins <address@hidden> wrote:
>
>> Hi,
>>
>> On Mon, March 28, 2016 10:56 am, Derek Atkins wrote:
>> > Hi,
>> >
>> > Thank you for taking the time to look at this..
>> >
>> > On Mon, March 28, 2016 10:41 am, Dominic Raferd wrote:
>> >> Is this really your first rdiff-backup to this location? If you have
>> any
>> >> previous rdiff-backup runs to this repository then the situation is
>> >> complicated by rdiff-backup's need to create a new set of reverse
>> diff
>> >> files to be able to regress to previous file contents.
>> >
>> > Yes, this s really the first rdiff-backup to this location.
>> >
>> > A second backup run shortly after the first one completed finished in
>> 55
>> > minutes.
>> >
>> >> What is your /tmp location? rdiff-backup uses this location for some
>> >> operations though not AFAIK for standard backup runs. Still, if /tmp
>> is
>> >> on
>> >> encfs maybe it could be a culprit; you can override rdiff-backup's
>> >> temporary file location with --tempdir and --remote-tempdir.
>> >
>> > If it is truly /tmp then no; /tmp is a ramdisk on the backup server
>> and
>> is
>> > on the root disk on the target server.  Neither are being run through
>> > encfs.
>> >
>> > If, however, you mean the rdiff-backup-data.tmp files, those ARE being
>> run
>> > through encfs.
>> >
>> >> Might also be worth trying --ssh-no-compression.
>> >
>> > I already have "Compression no" set in ~/.ssh/config so I'm not sure
>> what
>> > this would add?
>> >
>> >> Dominic
>> >> http://www.timedicer.co.uk
>> >
>> > -derek
>> >
>> > PS: I'm running a raw rsync command now just to see how it behaves --
>> so
>> > far I'm only seeing about 2MB/s, but it's only been running for 10
>> minutes
>> > or so.
>>
>> My rsync backup just finished.  It copied 202688912 KB in 1599m55.255s
>> so
>> about 2.1MB/s.  Still significantly slower than SCP, but faster than
>> rdiff-backup.
>>
>> The command I ran was:
>>
>> rsync -art -e "ssh -l root -i /dev/null -o Compression=no"
>> address@hidden:/var/www/ /backups/server
>>
>> :-(
>>
>> -derek
>>
>> >
>> >>
>> >> On 28 March 2016 at 14:37, Derek Atkins <address@hidden> wrote:
>> >>
>> >>> Just a quick update:  I tried making these changes on both sides and
>> it
>> >>> really didn't make a difference.  Full backup of 202852072 Kbytes
>> >>> required 2267m25.913s (previously it took 2346m57.800s, so it only
>> sped
>> >>> up a factor of 3%.
>> >>>
>> >>> Only thing I have not yet tried is running a raw rsync to see how
>> fast
>> >>> that runs.  I'll do that next.
>> >>>
>> >>> So, back to my orignal question: anyone have any idea how to get
>> >>> initial
>> >>> transfers to run faster (or indeed any significant data transfers)?
>> >>>
>> >>> Thanks,
>> >>>
>> >>> -derek
>> >>>
>> >>> "Derek Atkins" <address@hidden> writes:
>> >>>
>> >>> > Hi,
>> >>> >
>> >>> > I'm trying to use rdiff-backup to backup a bunch of servers.  One
>> >>> > particular server contains about 160GB of data, but when I try to
>> >>> perform
>> >>> > the rdiff-backup it's saving the data at a measly 1MB/s.
>> >>> >
>> >>> > Here's my configuration:
>> >>> >
>> >>> >   [server] <--ssh--> [backup-server]{encfs} <--nfs--> [freenas]
>> >>> >
>> >>> > I ran a bunch of tests to try to figure out my bottlenecks.
>> >>> >
>> >>> > I ran a bunch of dd tests (using dd if=/dev/zero bs=1M count=1000)
>> on
>> >>> the
>> >>> > backup server.  Going directly to FreeNAS via NFS (bybassing
>> encfs) I
>> >>> get
>> >>> > 50.2MB/s.  If I run dd directly on the backup server (through
>> encfs)
>> >>> I
>> >>> get
>> >>> > 20.1MB/s.  If I go over SSH from the backup server to the target
>> >>> server
>> >>> > and run the dd on the target server, then write to FreeNas through
>> >>> encfs
>> >>> > declines to 7.6MB/s.
>> >>> >
>> >>> > Note that in those SSH tests, however, I forgot to turn off
>> >>> compression.
>> >>> > When I do that, the throughput for the dd test reduced to 6.6BM/s.
>> >>> (Note
>> >>> > that this is running simultaneously with a running rdiff-backup,
>> so
>> >>> it's
>> >>> > possible that they are reducing performance).
>> >>> >
>> >>> > Then I ran an scp test to the same target server; copying about
>> 1.4GB
>> >>> of
>> >>> > photos.  Files ranged in size from 10KB to 5MB.  When run in
>> standard
>> >>> mode
>> >>> > (displaying each file status) I got 4.4MB/s.  Running in quiet
>> mode I
>> >>> get
>> >>> > 5.1MB/s.
>> >>> >
>> >>> > So clearly the bottleneck is in rdiff-backup -- performance (IMHO)
>> >>> should
>> >>> > not be significantly slower than the last dd-over-ssh test.  It
>> >>> appears
>> >>> > rdiff-backup is slowing me down by a factor of 5x throughput
>> versus
>> >>> scp.
>> >>> >
>> >>> > I found a message from Ben from 2005 where he suggests increasing
>> the
>> >>> > blocksize and conn_bufsiz settings in Globals.py:
>> >>> >
>> >>>
>> https://lists.gnu.org/archive/html/rdiff-backup-users/2005-10/msg00062.html
>> >>> >
>> >>> > What he didn't say was whether this needed to be changed on the
>> >>> target
>> >>> > server, the backup server, or both.  Nor do I know if that would
>> >>> actually
>> >>> > help this situation.
>> >>> >
>> >>> > Do you have any ideas?
>> >>> >
>> >>> > Thanks,
>> >>> >
>> >>> > -derek
>> >>> >
>> >>> > PS: According to rpm, both systems are running version 1.2.8.
>> >>>
>> >>> --
>> >>>        Derek Atkins                 617-623-3745
>> >>>        address@hidden             www.ihtfp.com
>> >>>        Computer and Internet Security Consultant
>> >>>
>> >>> _______________________________________________
>> >>> rdiff-backup-users mailing list at address@hidden
>> >>> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
>> >>> Wiki URL:
>> >>> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>> >>>
>> >>
>> >
>> >
>> > --
>> >        Derek Atkins                 617-623-3745
>> >        address@hidden             www.ihtfp.com
>> >        Computer and Internet Security Consultant
>> >
>> >
>> > _______________________________________________
>> > rdiff-backup-users mailing list at address@hidden
>> > https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
>> > Wiki URL:
>> > http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki
>> >
>>
>>
>> --
>>        Derek Atkins                 617-623-3745
>>        address@hidden             www.ihtfp.com
>>        Computer and Internet Security Consultant
>>
>>
>


-- 
       Derek Atkins                 617-623-3745
       address@hidden             www.ihtfp.com
       Computer and Internet Security Consultant