Re: [Duplicity-talk] duplicity collection status slowness

[Erik Romijn]

Hello all,

Thanks for all your suggestions. I've disabled GPG compression to exclude that 
as a factor (seems rather unnecessary anyways as we do gzip), but due to other 
reasons I started a new backup set which means it'll be a little while before 
the problem likely comes back. Once it does, I now have some better ideas of 
where to look  :)

Erik

On 26 Aug 2014, at 10:54, address@hidden wrote:

> i suggest you dig away.. 19 x 92 backups sounds like a lot of backups to me. 
> 
> on the other hand, just checked: my 3 monthly fulls with daily incrementals 
> (roughly 3x30) do a collection status in 6 seconds.
> 
> 1. maybe you sftp backend is just very slow. what happens when you delete the 
> archived cache?
> 2. does your system start swapping during listing, maybe because your vm has 
> very little ram?
> 
> ..ede/duply.net
> 
> 
> On 25.08.2014 18:19, Erik Romijn wrote:
>> Hello all,
>> 
>> Is there anyone who might be able to provide any insights regarding this 
>> issue? I'm happy to dig into the code myself, but I don't really know what 
>> would be the best place to start.
>> 
>> cheers,
>> Erik
>> 
>> On 20 Jul 2014, at 19:39, Erik Romijn <address@hidden> wrote:
>> 
>>> Hello all,
>>> 
>>> I'm using duplicity to run a few backups for my servers, and generally 
>>> found it to work very well. However, although my data is incredibly tiny, 
>>> duplicity has become incredibly slow, which I think I've narrowed down to 
>>> the collection status process.
>>> 
>>> My source file size is only 20MB, but running this backup takes about 7 
>>> minutes, and is almost completely cpu bound. Running the collection status 
>>> takes nearly the same amount, so it would seem that this is where the 
>>> slowness comes from.
>>> 
>>> I make incremental backups every 15 minutes, with a full backup after 23 
>>> hours, so 92 sets per day. I currently have 19 backup chains, according to 
>>> collection status, and there are no orphaned or incomplete sets.  The 
>>> source file size is about 20MB. In total the destination volume is 154MB 
>>> now. Running verify confirms that the backups are correct.
>>> 
>>> These numbers are for the backups of my /var/log, but I have another backup 
>>> of an unrelated directory of about 300MB on the same backup schema, which 
>>> shows similar numbers for collection status.
>>> 
>>> One workaround would be for me to move the files away from the duplicity 
>>> destination, so that the total collection appears smaller. But that leaves 
>>> me to wonder: why does collection status take so much time, particularly 
>>> considering it's cpu bound?
>>> 
>>> I'm running duplicity 0.6.23 with python 2.7.6 on an Ubuntu 14.04 VPS.
>>> 
>>> The full duplicity command line I use is:
>>> /usr/bin/duplicity --full-if-older-than 23h --encrypt-sign-key [...] 
>>> --verbosity info --ssh-options=-oIdentityFile=/root/.ssh/backup_rsa 
>>> --exclude-globbing-filelist /root/duplicity_log_exclude_filelist.txt 
>>> /var/log sftp://address@hidden/[...]/backups/log
>>> 
>>> Can anyone here provide insights into what might be the issue, and what 
>>> would be the best approach to tackle this?
>>> 
>>> cheers,
>>> Erik
>>> 
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk