[rdiff-backup-users] Re: Ways to speed rdiff-backup a bit

[Chris G]

On Thu, Jan 14, 2010 at 06:18:33PM +0000, Chris G wrote:
> I'm running rdiff-backup on a Western Digital My Book World Edition
> II, it's a little NAS server that runs linux and has ssh login
> available. 
> 
> The rdiff-backup is running from one of the NAS server's drives to the
> other drive, i.e. it's not being run from another system to (or from)
> the NAS and thus is not limited by network bandwidth.
> 
> I'm backing up two large[ish] directories, one is around 164Gb, the
> other is about 35Gb.
> 
> The 35Gb one takes about 7 minutes to run, which is fine, but the
> 165Gb one takes 4 hours and 5 minutes.  Presumably there's some sort
> of resource limitation which the bigger backup is hitting and hence
> running slower.
> 
> Is there anything I can do to improve performance?  My first guess
> would be that rdiff-backup is running out of 'real' memory and
> swapping which is slowing things.
> 
> It has 126828 kB of memory, more than I thought, that's 128Mb.
> 
> The processor is an ARM926EJ-S, 183 BogoMIPS.
> 
... and here are the rdiff-backup session_statistics files:-

StartTime 1263436518.00 (Thu Jan 14 02:35:18 2010)
EndTime 1263451841.07 (Thu Jan 14 06:50:41 2010)
ElapsedTime 15323.07 (4 hours 15 minutes 23.07 seconds)
SourceFiles 253324
SourceFileSize 167143773023 (156 GB)
MirrorFiles 253050
MirrorFileSize 167116487371 (156 GB)
NewFiles 274
NewFileSize 23594577 (22.5 MB)
DeletedFiles 0
DeletedFileSize 0 (0 bytes)
ChangedFiles 431
ChangedSourceSize 18543551628 (17.3 GB)
ChangedMirrorSize 18539860553 (17.3 GB)
IncrementFiles 706
IncrementFileSize 90764595 (86.6 MB)
TotalDestinationSizeChange 118050247 (113 MB)
Errors 0


StartTime 1263451844.00 (Thu Jan 14 06:50:44 2010)
EndTime 1263452264.06 (Thu Jan 14 06:57:44 2010)
ElapsedTime 420.06 (7 minutes)
SourceFiles 25405
SourceFileSize 35810108114 (33.4 GB)
MirrorFiles 25405
MirrorFileSize 35810108114 (33.4 GB)
NewFiles 0
NewFileSize 0 (0 bytes)
DeletedFiles 0
DeletedFileSize 0 (0 bytes)
ChangedFiles 0
ChangedSourceSize 0 (0 bytes)
ChangedMirrorSize 0 (0 bytes)
IncrementFiles 0
IncrementFileSize 0 (0 bytes)
TotalDestinationSizeChange 0 (0 bytes)
Errors 0



Hmmm, there's something funny going on here!  There should be some
changed files in that second (33Gb) backup, I'll go and investigate.

-- 
Chris Green