Re: [rdiff-backup-users] rdiff-backup streaming?

[mark price]

So if I have a 10 gig file backed up and want to check for updates and there is 
an update of only 500megs will it do this (which is what I need):

Read a block (let's say 100kb), then generate a signature for THAT block then 
generate a delta file for THAT block and then encrypt and compress THAT block, 
all in one streaming pass? And if so how is this performed? Is there some type 
of buffer function in the rdiff python wrapper?




On Feb 17, 2011, at 12:01 PM, Dominic Raferd <address@hidden> wrote:

> I used rdiff.exe briefly before moving to rdiff-backup. Certainly 
> rdiff-backup does your 2-4 'under the hood' and fast. It is highly optimised 
> both for speed of transfer and for storage space - but does require a 
> reliable connection between source and destination. It does not do your 1, 
> you can achieve this best by backing up from an LVM snapshot (if your source 
> machine is Linux) or VSS (if it is Windows).
> 
> I am pretty sure that rdiff-backup does not use rdiff, instead it uses a 
> python module built from the librsync library (which is also called, I think, 
> by rdiff).
> 
> Dominic
> http://www.timedicer.co.uk/
> 
> On 17/02/2011 17:41, Mark Price wrote:
>> Question -
>> 
>> We use rdiff (not rdiff-backup) to do our incremental file backups.
>> 
>> We do:
>> 
>>   1. Copy the file to a staging area (so the file won't disappear or
>>      be modified while we work on it)
>>   2. Hash the original file, and computes an rdiff signature (used
>>      for delta differencing)
>>   3. Comput an rdiff delta difference (if we have no prior version,
>>      this step is skipped)
>>   4. Compress & encrypt the resulting delta difference
>> 
>> Our problem is all of these things happen in separate phases, distinctly one 
>> from the other.  This means it takes a long time to do its job.  What I am 
>> wondering is if rdiff-backup does all of these things in one read/write file 
>> pass in a streaming manner, or if it simply calls to the standard rdiff.exe 
>> (which isn't working for us)?  I didn't quite see information concerning 
>> this in the docs or wiki.  We have considered using xdelta (because it 
>> operates in a streaming manner) but the problem with xdelta is that it 
>> stores double copies of the deltas and kills storage space.  Any help on 
>> this matter would be great!