Re: [Duplicity-talk] recover from aborted backup

[sophana]

Travis H. a écrit :
> On Tue, Nov 07, 2006 at 08:10:10AM +0100, sophana wrote:
>   
>> This patch works fine, broken transfers are restarted. But it does not
>> prevent data corruption.
>>     
>
> Most protocols do not, since it is handled at the TCP/IP layer.
>
> The checksums are not perfect (changing 0xFFFF to 0x0000 will pass
> the check due to one's-complement arithmetic), but it usually works.
>
>   
Some ftp servers now have the XCRC or XMD5 command to verify integrity.
This would be very nice to have.
Rsync is reliable because it is checksum based.
>> I think that when the connection is bad, and you need to restart the
>> transfer multiple times, it is probable that the data gets corrupted
>> (with ftp).
>>     
>
> Are you using vsftpd by any chance?  Because I've found what apparently
> are race conditions in that server which lead to non-deterministic
> data corruption.
>
> I've been meaning to write a "add redundancy to any file" filter at
> some point with lots of configuration options.  There's plenty of tools
> which do CRCs anyway, but it's usually also required to put it in a
> particular format, which is not very "Unixy".
>   

Redundancy is also a good idea. The standard tool I have in mind is the
one used for binary newsgroups: par files allows that some archives are
missing, it can reconstruct the missing data. Associated with a multiple
server storage, you get efficient redundant backup.
This also would be very nice to have.
Dibs distributed backup system has such similar features.