Re: [Duplicity-talk] Re: Duplicity using more space than needed?

[Kenneth Loafman]

OK, the problem is fixed and in CVS (diffdir.py).  I'll have it in the 
next RC once I have some other fixes done.

The problem was indeed in the way we called librsync, namely the block 
size to use.  In the case of a 49MB file, the block size would have been 
24k, which means that every block in the test had a change, so every 
block got backed up on the incremental.  I set it to max out at 2k and 
it now gives the same performance as rdiff.  This increases the number 
of sig file entries and its size.  Runtime is only increased a small 
amount, so I think the tradeoffs are acceptable.

The best response may be to allow the user to set the maximum block size 
via an option since some files only change at the end of the file.

...Ken

Fabiano Bonin wrote:
> Ok, count on me if you need to do some testing.
>
> Unfortunatelly i don't code in python, otherwise i would be glad to help.
>
> Regards,
>
> Fabiano.
>
> On 6/29/07, Kenneth Loafman <address@hidden> wrote:
> > Thanks for the testcase.  I'm going to add it to the tests in the
> > repository, once I get this fixed.
> >
> > Both duplicity and rdiff use librsync for deltas, so the problem has to
> > be in that we handle the parameters or the calls differently than rdiff.
> >
> > ...Ken
> >
> > Fabiano Bonin wrote:
> > > Look at this test case. I think it reproduces the problem.
> > >
> > > First, run the script 'testcase_script.sh' attached to this message.
> > > It will create the following files in /var/tmp/testcase:
> > >
> > > snapshot1 (48MB)
> > > snapshot2 (48MB)
> > > snapshot1.sig (287KB)
> > > snapshot2.delta (20MB)
> > >
> > > Comment: for each block of 5000 bytes of snapshot1, i changed the last
> > > 25 bytes in snapshot2. All bytes where generated randomly.
> > >
> > > snapshot2.delta is the uncompressed rdiff delta file between snapshot1
> > > and snapshot2.
> > >
> > > Then, run the following commands:
> > >
> > > ==============================
> > > export PASSPHRASE=testcase
> > >
> > > rm /var/tmp/testcase_source/* -rf
> > > rm /var/tmp/testcase_destination/* -rf
> > >
> > > cp snapshot1 /var/tmp/testcase_source/snapshot
> > >
> > > duplicity /var/tmp/testcase_source file:///var/tmp/testcase_destination
> > >
> > > du --max-depth 1 -h /var/tmp/testcase_destination
> > >
> > > rm -f /var/tmp/testcase_source/snapshot
> > >
> > > cp snapshot2 /var/tmp/testcase_source/snapshot
> > >
> > > duplicity /var/tmp/testcase_source file:///var/tmp/testcase_destination
> > >
> > > du --max-depth 1 -h /var/tmp/testcase_destination
> > > ==============================
> > >
> > > Here, in the second pass, the destination size change is 47.9MB. The
> > > last 'du' command will show the folder /var/tmp/testcase_destination
> > > with 97MB.
> > >
> > > Regards,
> > >
> > > Fabiano.
> > >
> > >
> > > 
> > ------------------------------------------------------------------------
> > >
> > > _______________________________________________
> > > Duplicity-talk mailing list
> > > address@hidden
> > > http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> >
> >
> > _______________________________________________
> > Duplicity-talk mailing list
> > address@hidden
> > http://lists.nongnu.org/mailman/listinfo/duplicity-talk
>
>
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk