Re: [Duplicity-talk] Duplicity Backup - check files uploaded to a backend against corruption using a checksum ?

[edgar . soldin]

let's place this topic on the mailing list for others to find, shall we ;)

Kostas,

we already use checksums. "un"fortunately they are encrypted, in the manifest i 
think, so actual distorted gpg files will cause a hickup with gpg decryption 
even before duplicity can detect any corruption.

wrt. to your approach. i'd rather have a more universal (woking with all 
backends) one, like the par2 backend
http://bazaar.launchpad.net/~germar/duplicity/par2/revision/920

i am not sure what the status is on it though. Germar?

..ede/duply.net


On 31.01.2014 23:51, Kostas Papadopoulos wrote:
> Hi folks,
> 
> A feature which seems interesting, but I'm not quite sure if/how it could fit 
> into duplicity, would be to *check the files uploaded to the backend* 
> *against corruption using a checksum* (in case of GoogleDrive a MD5 checksum 
> offered via the API). I realise that it doesn't totally fit into duplicity's 
> "any dumb backend" design, but a simple filesize+md5 would catch most errors 
> ...
> 
> I actually do this sort of check on any files I put on GoogleDrive via the 
> Drive v2 API using the OAuth 2.0 Playground
> 
>        "originalFilename": "backup",
>        "fileExtension": "",
>        *"md5Checksum"**:** **"502e74a09ff18efa312a70427e613f97"**,*
>        "fileSize": "67108864",
>        "quotaBytesUsed": "67108864",
> 
> 
> And here's Google's take on it:
> 
>     /I would not worry about this. We can't share the specifics but data 
> hosted on Google is checked against corruption and is also replicated 
> multiple times./
> 
>     //
> 
>     /This doesn't prevent you from uploading corrupted data though. So //*you 
> could potentially use the read-only MD5 checksum field post upload to make 
> sure that the file that you just uploaded to Drive has the correct MD5 if 
> data consistency is critical for you*//.//
>     /
> 
>     
> /http://stackoverflow.com/questions/10328992/is-data-corruption-on-a-google-data-server-automatically-detected/
> 
> 
> On 1/3/2014 7:23 AM, Kenneth Loafman wrote:
>> Thanks guys!  Sorry for the confusion.
>>
>>
>> On Fri, Jan 3, 2014 at 4:40 AM, <address@hidden <mailto:address@hidden>> 
>> wrote:
>>
>>     recommitted completely just now :)..
>>     Kostas: thanks for paying close attention..
>>
>>     ..ede
>>
>>     On 02.01.2014 06:02, Kostas Papadopoulos wrote:
>>     > The committ at launchpad seems to be missing one line (after line 
>> #222):
>>     >
>>     >     [...]
>>     >                     for entry in entries:
>>     >                         *resource_type = entry.get_resource_type()*
>>     >                         if (not type) or (type == 'folder' and 
>> resource_type == 'folder') or (type == GDocsBackend.BACKUP_DOCUM
>>     >     ENT_TYPE and resource_type != 'folder'):
>>     >
>>     >
>>     > On 1/1/2014 11:25 AM, address@hidden <mailto:address@hidden> wrote:
>>     >> thanks.. committed.. ede
>>     >>
>>     >> On 29.12.2013 23:19, Kostas Papadopoulos wrote:
>>     >>> Hi,
>>     >>>
>>     >>> Since I noticed several fixes have been committed into the duplicity 
>> tree in the last couple of days, I'd just like to report back that for the 
>> past 40 days I've been running duplicity 0.6.22 with Carlos Abalde's patch 
>> <http://lists.nongnu.org/archive/html/duplicity-talk/2013-11/msg00017.html> 
>> to the gdocs backend.
>>     >>>
>>     >>> I'd like to thank you all for your work and wish you a happy New 
>> Year 2014,
>>     >>> KP
>>     >>>
>>     >>> On 11/20/2013 2:10 PM, Kostas Papadopoulos wrote:
>>     >>>> Hi,
>>     >>>>
>>     >>>> Following up Edgar Soldin's question at
>>     >>>> 
>> http://lists.nongnu.org/archive/html/duplicity-talk/2013-11/msg00017.html
>>     >>>>
>>     >>>> I'd like to report that Carlos Abalde's patch also seems to work 
>> for me.
>>     >>>>
>>     >>>> Best regards,
>>     >>>> KP
>>     >>>>
>>     >>>> On 11/19/2013 10:40 AM, Kenneth Loafman wrote:
>>     >>>>> I just applied the patch and pushed it to the repository.
>>     >>>>>
>>     >>>>> Thanks for the report!
>>     >>>>>
>>     >>>>> ...Ken
>>     >>>>>
>>     >>>>>
>>     >>>>>
>>     >>>>> On Mon, Nov 18, 2013 at 6:02 PM, Kostas Papadopoulos 
>> <address@hidden <mailto:address@hidden> <mailto:address@hidden 
>> <mailto:address@hidden>>> wrote:
>>     >>>>>
>>     >>>>>     Dear Kenneth,
>>     >>>>>
>>     >>>>>     Please note that when trying to use duplicity 0.6.22 and duply 
>> duply 1.5.11 with GoogleDocs backend (under Debian Wheezy), I experienced 
>> the same errors as described here:
>>     >>>>>
>>     >>>>>     
>> http://lists.nongnu.org/archive/html/duplicity-talk/2013-07/msg00007.html
>>     >>>>>
>>     >>>>>     I just applied his patch, and duplicity+duply now seem to work 
>> fine.
>>     >>>>>
>>     >>>>>     Best regards,
>>     >>>>>     KP
>>     >>>>>
>>     >>>>>
>>     >
>>
>>
>