[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Duplicity-talk] Duplicity full backup has become slow
From: |
edgar . soldin |
Subject: |
Re: [Duplicity-talk] Duplicity full backup has become slow |
Date: |
Sun, 12 Nov 2017 19:25:32 +0100 |
User-agent: |
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 |
On 11/12/2017 19:19, Matthew Harrison via Duplicity-talk wrote:
> Hi,
>
> I've been happy using duplicity to backup my Ubuntu PC to my NAS for around
> 18 months now, using rsync.
>
> I do a full back up every 2 weeks, and and incremental daily.
>
> I noticed in August that the full backup started taking around twice as
> long, from
>
> --------------[ Backup Statistics ]--------------
> StartTime 1502355166.95 (Thu Aug 10 09:52:46 2017)
> EndTime 1502373067.73 (Thu Aug 10 14:51:07 2017)
> ElapsedTime 17900.78 (4 hours 58 minutes 20.78 seconds)
> SourceFiles 1361618
> SourceFileSize 212181777813 (198 GB)
> NewFiles 1361618
> NewFileSize 212181777813 (198 GB)
> DeletedFiles 0
> ChangedFiles 0
> ChangedFileSize 0 (0 bytes)
> ChangedDeltaSize 0 (0 bytes)
> DeltaEntries 1361618
> RawDeltaSize 211041882531 (197 GB)
> TotalDestinationSizeChange 166597720163 (155 GB)
> Errors 0
> -------------------------------------------------
>
> to
>
> --------------[ Backup Statistics ]--------------
> StartTime 1503684459.99 (Fri Aug 25 19:07:39 2017)
> EndTime 1503718307.60 (Sat Aug 26 04:31:47 2017)
> ElapsedTime 33847.61 (9 hours 24 minutes 7.61 seconds)
> SourceFiles 1361798
> SourceFileSize 212042512113 (197 GB)
> NewFiles 1361798
> NewFileSize 212042512113 (197 GB)
> DeletedFiles 0
> ChangedFiles 0
> ChangedFileSize 0 (0 bytes)
> ChangedDeltaSize 0 (0 bytes)
> DeltaEntries 1361798
> RawDeltaSize 210902807446 (196 GB)
> TotalDestinationSizeChange 166567853868 (155 GB)
> Errors 0
> -------------------------------------------------
>
> As you can see, pretty much the same statistics except for the time taken.
>
> Now in between these 2 full backups, probably the most significant change
> to my system was an upgrade from Ubuntu 16.10 to 17.04.
>
> I don't think network speed is an issue because some of the (4GB) vol files
> can take as little as 5 minutes (looking at the timestamps) which I assume
> includes creation and transfer time.
> I've noticed that the slowest archive files to be produced took around 1h
> 25m, and either consisted of lots and lots of small files, or very large
> binary files.
>
> Is anyone aware of any change in the versions of the Ubuntu packages that
> duplicity relies on, between Ubuntu 16.10 and 17.04 that might have caused
> this slowdown?
>
> I haven't looked yet at adding extra logging to help see if my thoughts on
> creation vs transfer time are correct.
>
> On the plus side, this has nudged me to look more closely at what
> unnecessary cruft I'm backing up :D
>
> Many thanks, and best regards,
Matthew,
what are the respective duplicity versions that were/are used?
..ede/duply.net