[Duplicity-talk] Re: Duplicity using more space than needed?

[Fabiano Bonin]

Just to make the things worse, here is the comparision if i compact
the rdiff deltas:

SNAPSHOT1 -> SNAPSHOT2

gziped rdiff delta size - 37MB   duplicity dest size change - 151MB

SNAPSHOT2 -> SNAPSHOT3

gziped rdiff delta size - 4.7MB   duplicity dest size change - 106MB

Supposing my usual scenario is what happens in SNAPSHOT2 -> SNAPSHOT3,
if i want to keep 365 days of incremental backups, using rdiff deltas
i will need to buy +- 2GB of storage space, while using duplicity i
will need +- 40GB.

Taking rsync.net storage prices, it means a yearly amount of US$ 38.4
using rdiff against US$ 768,00 using duplicity.

On 6/21/07, Fabiano Bonin <address@hidden> wrote:
> Hi all,
>
> I spent some hours last days trying to reproduce what may be a
> duplicity wrong behavior or bug, and finally i have a test case.
>
> I will reproduce this test case for a duplicity expert to analyze.
>
> I started with these files:
>
> address@hidden tmp]# ll -h
> -rw------- 1 root root 1.7G Jun 19 14:33 tecelagem.fdb_19-06_14:30
>  <-------- SNAPSHOT1
> -rw-r--r-- 1 root root 9.9M Jun 20 08:36 tecelagem.fdb_19-06_14:30.sig
> -rw------- 1 root root 1.7G Jun 20 08:41 tecelagem.fdb_20-06_08:41
>  <-------- SNAPSHOT2
> -rw-r--r-- 1 root root 276M Jun 20 08:52 tecelagem.fdb_20-06_08:41.del
> -rw-r--r-- 1 root root 9.9M Jun 21 08:11 tecelagem.fdb_20-06_08:41.sig
> -rw------- 1 root root 1.7G Jun 20 16:32 tecelagem.fdb_20-06_16:30
>  <-------- SNAPSHOT3
> -rw-r--r-- 1 root root  13M Jun 21 08:12 tecelagem.fdb_20-06_16:30.del
> -rw-r--r-- 1 root root 9.9M Jun 21 08:25 tecelagem.fdb_20-06_16:30.sig
>
> The file 'tecelagem.fdb_19-06_14:30' is a copy i took from a database
> file named 'tecelagem.fdb' at 19/06, 14:30.
> I will call this file SNAPSHOT1.
> The file 'tecelagem.fdb_19-06_14:30.sig' is its signature, generated with rdiff.
>
> The file 'tecelagem.fdb_20-06_08:41' is a copy of the same file, took
> at 20/06, 08:41.
> I will call this file SNAPSHOT2.
> The file 'tecelagem.fdb_20-06_08:41.sig' is its signature, generated with rdiff.
> The file 'tecelagem.fdb_20-06_08:41.del' is the delta file generated
> against SNAPSHOT1 signature using rdiff.
>
> The file 'tecelagem.fdb_20-06_16:30' is a copy of the same file, took
> at 20/06, 16:30.
> I will call this file SNAPSHOT3.
> The file 'tecelagem.fdb_20-06_16:30.sig' is its signature, generated with rdiff.
> The file 'tecelagem.fdb_20-06_16:30.del' is the delta file generated
> against SNAPSHOT2 signature using rdiff.
>
> Notice that SNAPSHOT2 delta has 276MB and SNAPSHOT3 delta has 13MB. It
> was intentional, because i did a lot of modifications to the database
> file after taking SNAPSHOT1.
>
> So, here starts the test:
>
> address@hidden tmp]# mkdir /var/tmp/source_duplicity
> address@hidden tmp]# mkdir /var/tmp/destination_duplicity
>
> Copying SNAPSHOT1 as 'tecelagem.fdb' to /var/tmp/source_duplicity
>
> address@hidden tmp]# cp tecelagem.fdb_19-06_14:30
> /var/tmp/source_duplicity/tecelagem.fdb
>
> address@hidden tmp]# duplicity -v 5 ./source_duplicity
> file:///var/tmp/destination_duplicity
> --------------[ Backup Statistics ]--------------
> StartTime 1182426875.80 (Thu Jun 21 08:54:35 2007)
> EndTime 1182427114.55 (Thu Jun 21 08:58:34 2007)
> ElapsedTime 238.75 (3 minutes 58.75 seconds)
> SourceFiles 2
> SourceFileSize 1767665664 (1.65 GB)
> NewFiles 2
> NewFileSize 1767665664 (1.65 GB)
> DeletedFiles 0
> ChangedFiles 0
> ChangedFileSize 0 (0 bytes)
> ChangedDeltaSize 0 (0 bytes)
> DeltaEntries 2
> RawDeltaSize 5174784 (4.94 MB)
> TotalDestinationSizeChange 371100021 (354 MB)
> Errors 0
> -------------------------------------------------
>
> At this first run, duplicity makes a full backup. As you can see, it
> uses 354MB to store the backup.
>
> Copying SNAPSHOT2 as 'tecelagem.fdb' to /var/tmp/source_duplicity
>
> address@hidden tmp]# cp tecelagem.fdb_20-06_08:41
> /var/tmp/source_duplicity/tecelagem.fdb
>
> address@hidden tmp]# duplicity -v 5 ./source_duplicity
> file:///var/tmp/destination_duplicity
> --------------[ Backup Statistics ]--------------
> StartTime 1182427288.46 (Thu Jun 21 09:01:28 2007)
> EndTime 1182427467.48 (Thu Jun 21 09:04:27 2007)
> ElapsedTime 179.02 (2 minutes 59.02 seconds)
> SourceFiles 2
> SourceFileSize 1768116224 (1.65 GB)
> NewFiles 0
> NewFileSize 0 (0 bytes)
> DeletedFiles 0
> ChangedFiles 1
> ChangedFileSize 1768112128 (1.65 GB)
> ChangedDeltaSize 0 (0 bytes)
> DeltaEntries 1
> RawDeltaSize 5174784 (4.94 MB)
> TotalDestinationSizeChange 158668541 (151 MB)
> Errors 0
> -------------------------------------------------
>
> At the second run, duplicity makes an incremental backup, and uses
> 151MB extra space to store the delta. Remember that rdiff generated a
> uncompressed 276MB delta in this snapshot.
>
> Copying SNAPSHOT3 as 'tecelagem.fdb' to /var/tmp/source_duplicity
>
> address@hidden tmp]# cp tecelagem.fdb_20-06_16:30
> /var/tmp/source_duplicity/tecelagem.fdb
>
> address@hidden tmp]# duplicity -v 5 ./source_duplicity
> file:///var/tmp/destination_duplicity
> --------------[ Backup Statistics ]--------------
> StartTime 1182427586.75 (Thu Jun 21 09:06:26 2007)
> EndTime 1182427714.27 (Thu Jun 21 09:08:34 2007)
> ElapsedTime 127.51 (2 minutes 7.51 seconds)
> SourceFiles 2
> SourceFileSize 1768116224 (1.65 GB)
> NewFiles 0
> NewFileSize 0 (0 bytes)
> DeletedFiles 0
> ChangedFiles 1
> ChangedFileSize 1768112128 (1.65 GB)
> ChangedDeltaSize 0 (0 bytes)
> DeltaEntries 1
> RawDeltaSize 5174784 (4.94 MB)
> TotalDestinationSizeChange 111287108 (106 MB)
> <-------------------------------- Here is the problem
> Errors 0
> -------------------------------------------------
>
> At the third run, duplicity makes an incremental backup, and uses
> 106MB! extra space to store the delta, and here is the problem. With
> rdiff, the delta file between SNAPSHOT2 and SNAPSHOT3 is just 13MB.
>
> Is there some reason for this to happen?
>
> Regards,
>
> Fabiano.