Re: [Duplicity-talk] Duplicity Backup - check files uploaded to a backend against corruption using a checksum ?

[edgar . soldin]

Germar,

i'd like you to rename the backend file to ~par2wrapperbackend.py to make sure 
that a theoretical future zzzbackend.py would be included as well.

also could you please add documentation to the manpage in sections
Requirements
URL format
while updating the branch?

thanks!.. ede/duply.net

On 01.02.2014 21:14, Germar Reitze wrote:
> Hi all together,
> 
> par2-backend is working fine. I'll merge current trunk into par2-backend
> soon and it will need some more testing. I just stopped using it couple
> month ago because of https://bugs.launchpad.net/duplicity/+bug/385495
> 
> Just drop me a line if you want me to fill a merge request.
> 
> Regards,
> Germar
> 
> Am Samstag, den 01.02.2014, 18:34 +0100 schrieb address@hidden:
>> let's place this topic on the mailing list for others to find, shall we ;)
>>
>> Kostas,
>>
>> we already use checksums. "un"fortunately they are encrypted, in the 
>> manifest i think, so actual distorted gpg files will cause a hickup with gpg 
>> decryption even before duplicity can detect any corruption.
>>
>> wrt. to your approach. i'd rather have a more universal (woking with all 
>> backends) one, like the par2 backend
>> http://bazaar.launchpad.net/~germar/duplicity/par2/revision/920
>>
>> i am not sure what the status is on it though. Germar?
>>
>> ..ede/duply.net
>>
>>
>> On 31.01.2014 23:51, Kostas Papadopoulos wrote:
>>> Hi folks,
>>>
>>> A feature which seems interesting, but I'm not quite sure if/how it could 
>>> fit into duplicity, would be to *check the files uploaded to the backend* 
>>> *against corruption using a checksum* (in case of GoogleDrive a MD5 
>>> checksum offered via the API). I realise that it doesn't totally fit into 
>>> duplicity's "any dumb backend" design, but a simple filesize+md5 would 
>>> catch most errors ...
>>>
>>> I actually do this sort of check on any files I put on GoogleDrive via the 
>>> Drive v2 API using the OAuth 2.0 Playground
>>>
>>>        "originalFilename": "backup",
>>>        "fileExtension": "",
>>>        *"md5Checksum"**:** **"502e74a09ff18efa312a70427e613f97"**,*
>>>        "fileSize": "67108864",
>>>        "quotaBytesUsed": "67108864",
>>>
>>>
>>> And here's Google's take on it:
>>>
>>>     /I would not worry about this. We can't share the specifics but data 
>>> hosted on Google is checked against corruption and is also replicated 
>>> multiple times./
>>>
>>>     //
>>>
>>>     /This doesn't prevent you from uploading corrupted data though. So 
>>> //*you could potentially use the read-only MD5 checksum field post upload 
>>> to make sure that the file that you just uploaded to Drive has the correct 
>>> MD5 if data consistency is critical for you*//.//
>>>     /
>>>
>>>     
>>> /http://stackoverflow.com/questions/10328992/is-data-corruption-on-a-google-data-server-automatically-detected/
>>>
>>>
>>> On 1/3/2014 7:23 AM, Kenneth Loafman wrote:
>>>> Thanks guys!  Sorry for the confusion.
>>>>
>>>>
>>>> On Fri, Jan 3, 2014 at 4:40 AM, <address@hidden <mailto:address@hidden>> 
>>>> wrote:
>>>>
>>>>     recommitted completely just now :)..
>>>>     Kostas: thanks for paying close attention..
>>>>
>>>>     ..ede
>>>>
>>>>     On 02.01.2014 06:02, Kostas Papadopoulos wrote:
>>>>     > The committ at launchpad seems to be missing one line (after line 
>>>> #222):
>>>>     >
>>>>     >     [...]
>>>>     >                     for entry in entries:
>>>>     >                         *resource_type = entry.get_resource_type()*
>>>>     >                         if (not type) or (type == 'folder' and 
>>>> resource_type == 'folder') or (type == GDocsBackend.BACKUP_DOCUM
>>>>     >     ENT_TYPE and resource_type != 'folder'):
>>>>     >
>>>>     >
>>>>     > On 1/1/2014 11:25 AM, address@hidden <mailto:address@hidden> wrote:
>>>>     >> thanks.. committed.. ede
>>>>     >>
>>>>     >> On 29.12.2013 23:19, Kostas Papadopoulos wrote:
>>>>     >>> Hi,
>>>>     >>>
>>>>     >>> Since I noticed several fixes have been committed into the 
>>>> duplicity tree in the last couple of days, I'd just like to report back 
>>>> that for the past 40 days I've been running duplicity 0.6.22 with Carlos 
>>>> Abalde's patch 
>>>> <http://lists.nongnu.org/archive/html/duplicity-talk/2013-11/msg00017.html>
>>>>  to the gdocs backend.
>>>>     >>>
>>>>     >>> I'd like to thank you all for your work and wish you a happy New 
>>>> Year 2014,
>>>>     >>> KP
>>>>     >>>
>>>>     >>> On 11/20/2013 2:10 PM, Kostas Papadopoulos wrote:
>>>>     >>>> Hi,
>>>>     >>>>
>>>>     >>>> Following up Edgar Soldin's question at
>>>>     >>>> 
>>>> http://lists.nongnu.org/archive/html/duplicity-talk/2013-11/msg00017.html
>>>>     >>>>
>>>>     >>>> I'd like to report that Carlos Abalde's patch also seems to work 
>>>> for me.
>>>>     >>>>
>>>>     >>>> Best regards,
>>>>     >>>> KP
>>>>     >>>>
>>>>     >>>> On 11/19/2013 10:40 AM, Kenneth Loafman wrote:
>>>>     >>>>> I just applied the patch and pushed it to the repository.
>>>>     >>>>>
>>>>     >>>>> Thanks for the report!
>>>>     >>>>>
>>>>     >>>>> ...Ken
>>>>     >>>>>
>>>>     >>>>>
>>>>     >>>>>
>>>>     >>>>> On Mon, Nov 18, 2013 at 6:02 PM, Kostas Papadopoulos 
>>>> <address@hidden <mailto:address@hidden> <mailto:address@hidden 
>>>> <mailto:address@hidden>>> wrote:
>>>>     >>>>>
>>>>     >>>>>     Dear Kenneth,
>>>>     >>>>>
>>>>     >>>>>     Please note that when trying to use duplicity 0.6.22 and 
>>>> duply duply 1.5.11 with GoogleDocs backend (under Debian Wheezy), I 
>>>> experienced the same errors as described here:
>>>>     >>>>>
>>>>     >>>>>     
>>>> http://lists.nongnu.org/archive/html/duplicity-talk/2013-07/msg00007.html
>>>>     >>>>>
>>>>     >>>>>     I just applied his patch, and duplicity+duply now seem to 
>>>> work fine.
>>>>     >>>>>
>>>>     >>>>>     Best regards,
>>>>     >>>>>     KP
>>>>     >>>>>
>>>>     >>>>>
>>>>     >
>>>>
>>>>
>>>
>>
>