Re: [Duplicity-talk] What is Duplicity doing during incremental backup?

[Nicholas Sherlock]

Okay, at 400 minutes it was still doing the same thing, so I killed
that task and restarted. Now it quickly completes:

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Tue Feb 21 13:03:05 2012
python: ERROR: (rs_job_complete) loadsig job failed: out of memory
Error librsync error 102 while in delta rs_signature_t builder getting
delta for lib/mysql/ibdata1
--------------[ Backup Statistics ]--------------
StartTime 1329902068.63 (Wed Feb 22 09:14:28 2012)
EndTime 1329902563.24 (Wed Feb 22 09:22:43 2012)
ElapsedTime 494.61 (8 minutes 14.61 seconds)
SourceFiles 796
SourceFileSize 412831898 (394 MB)
NewFiles 0
NewFileSize 0 (0 bytes)
DeletedFiles 0
ChangedFiles 30
ChangedFileSize 143144207 (137 MB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 30
RawDeltaSize 134604624 (128 MB)
TotalDestinationSizeChange 51429054 (49.0 MB)
Errors 1
-------------------------------------------------

However, it seems to have run out of memory during the job. How can I
estimate how much memory and temporary disk space will be needed for a
given backup process? There seems to be no information available about
resource requirements for Duplicity which is rather problematic for
large backups.

Cheers,
Nicholas Sherlock

On Wed, Feb 22, 2012 at 1:41 PM, Nicholas Sherlock
<address@hidden> wrote:
>
> Hi everyone,
>
> I'm trying to backup a copy of my database to RackSpace cloud file storage 
> using duplicity 0.6.15. The initial full backup went fine:
>
> Now I'm trying to re-run duplicity in order to create an incremental backup. 
> However, it has been running for a long time and doesn't seem to be doing 
> very much. No new files from the incremental backup have appeared in 
> Rackspace. iostat shows that nothing is >happening on disk (nothing is being 
> read from the source device, /dev/md127):
>
>
> Duplicity seems to be very busy doing something, using 99.5% of the CPU and 
> 86.7% of the memory, looks like it has been running for 2 hours now:
>
>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>  1129 root      20   0 1455m 1.4g 4716 R 99.5 86.7 126:05.34 duplicity
>
> Is that normal? What the heck is it doing? Could I speed up this process by 
> running it on a multi-core box?
>