Re: [Duplicity-talk] Access GDrive through gdocs backend failing. Link to patch.

[edgar . soldin]

On 05.07.2013 05:09, Boris ARZUR wrote:
>> Message: 2
>> Date: Thu, 04 Jul 2013 11:55:53 +0200
>> From: address@hidden
>> To: Discussion of the backup program duplicity
>>      <address@hidden>
>> Subject: Re: [Duplicity-talk] Access GDrive through gdocs backend
>>      failing. Link to patch.
>> Message-ID: <address@hidden>
>> Content-Type: text/plain; charset=ISO-8859-1
>>
>> On 04.07.2013 09:10, Boris ARZUR wrote:
>>> Hello everyone,
>>>
>>> First, let me thank you for a wonderful utility. I love duplicity !
>>>
>>> I use duplicity 0.6.21. I have been playing with 'big' backups on google
>>> drive. I use the gdocs backend. The full backup works like a charm. And my
>>> troubles began with the incremental.
>>>
>>> Downloading the metadata was failing with :
>>>
>>> Failed to download file
>>> 'duplicity-full-signatures.20130703T020608Z.sigtar'
>>> in remote folder 'New Folder': Failed to find file
>>> 'duplicity-full-signatures.20130703T020608Z.sigtar'
>>> in remote folder 'New Folder'
>>> see : http://pastebin.com/JbTNTh0d for more details.
>>>
>>> duplicity-full-signatures.20130703T020608Z.sigtar is 188 MB.
>>> This problem does not happen with smaller files (< 20 MB).
>>>
>>> I tracked the problem to a check in
>>> duplicity/backends/gdocsbackend.py: GDocsBackend.__fetch_entries
>>> where the entry.get_resource_type() turns out to be 'application/x-tar'
>>> and not 'application/binary' as expected...
>>>
>>> I put together a quick fix, you can find it here :
>>> http://pastebin.com/V5anwj1M
>>> [basically test only on the left side of the /, enough to
>>> distinguish folders from files]
>>>
>>> It works for me. I hope it can be of use in duplicity.
>>>
>>> Have a good day. And thank you for the all the duplicates...
>>> Boris.
>>>
>>>
>>
>> not familiar with the backend code. 
>> but if you could explain a bit more why it failed and why it works now, what 
>> the code did before vs. now, i could setup a launchpad branch for the 
>> maintainer to merge.
>>
>> ..ede/duply.net
> 
> Hi,
> OK, I sorted out my mailman problems :)
> 
>> [...] explain a bit more why it failed and why it works now, [...]
>>> I tracked the problem to a check in
>>> duplicity/backends/gdocsbackend.py: GDocsBackend.__fetch_entries
>>> where the entry.get_resource_type() turns out to be 'application/x-tar'
>>> and not 'application/binary' as expected...
> As far as I understant, __fetch_entries is used to list directory and files
> that match a 'filename' pattern. It sends a request to google servers with a
> few parameters in the url (like 'title' or 'title-exact').
> The reply is parsed an iterable object containing 'Resource' objects.
> 
> When duplicity sends for a file s.a. 
> 'duplicity-full-signatures.20130703T020608Z.sigtar',
> it sets title=%(the_filename)s and title-exact=true in the url, sends out
> the request. 
> When the reply comes, it swifts through the results for a directory/filename
> match (it seems you cannot specify the working directory in the url) and for
> a resource type corresponding to a file, i.e. 'application/binary' here.
> 
> The problem seems to be caused by this last check. For big files, this check
> excludes all files, the result sets remains empty.
> 
> To illustrate the problem at hand, I made an account in gmail :
> address@hidden / pass : 1pnln04bteescuat60l31
> (account linked to address@hidden, 10 minutes mail)
> 
> If you run (be careful with tmp and tmp2, dont overwrite data :) ):
> <shell>
> $ mkdir tmp # create temp dir
> $ dd if=/dev/urandom of=tmp/blob bs=1M count=30 # create file, 30M random data
> # backup the tmp dir
> $ duplicity -v8 --no-encryption --volsize 150 --no-compression\
>    `pwd`/tmp \
>    gdocs://test.gdocs.backend.duplicity:address@hidden/tmp
> # and restore
> $ duplicity -v8 --no-encryption --volsize 150 --no-compression\
>    gdocs://test.gdocs.backend.duplicity:address@hidden/tmp\
>    `pwd`/tmp2 
> # this should fail with :
> # Attempt 1 failed: BackendException: Failed to download file
> # 'duplicity-full.20130705T023038Z.vol1.difftar' in remote folder 'tmp':
> # Failed to find file 'duplicity-full.20130705T023038Z.vol1.difftar' in
> # remote folder 'tmp'
> </shell>
> 
> You can monitor the 'failing' with :
> <patch>
> --- /usr/lib/python2.7/site-packages/duplicity/backends/gdocsbackend.orig.py 
> 2013-07-04 08:22:24.030257937 +0200
> +++ /usr/lib/python2.7/site-packages/duplicity/backends/gdocsbackend.py 
> 2013-07-05 04:40:40.599483189 +0200
> @@ -220,6 +220,7 @@
>              if title:
>                  result = []
>                  for entry in entries:
> +                    print "debug info in __fetch_entries: type, get_res_type 
> :", type, entry.get_resource_type()
>                      if (not type) or (entry.get_resource_type() == type):
>                          if folder_id != GDocsBackend.ROOT_FOLDER_ID:
>                              for link in entry.in_collections():
> </patch>
> 
> You get info like :
> [...]
> Import of duplicity.backends.hsibackend Succeeded
> debug info in __fetch_entries: type, get_res_type : folder folder
> Main action: restore
> [...]
> No orphaned or incomplete backup sets found.
> debug info in __fetch_entries: type, get_res_type : application/binary 
> application/x-gzip
> Attempt 1 failed: [...]
> 
>> [...] what the code did before vs. now [...]
> The test "if (not type) or (entry.get_resource_type() == type):" is made
> lest stringent. It just discriminates folders and files (matching
> 'application/binary' with 'application/*' and 'folder' with 'folder').
> 
> BTW, if you try after patching, please remove '--no-compression', it just
> doesnt work... another bug maybe :)
> 
> I used :
> <shell>
> $ PASSPHRASE='i_want+you+to+&duplicate_this' duplicity -v8 --volsize 150\
>    `pwd`/tmp \
>     gdocs://test.gdocs.backend.duplicity:address@hidden/tmp
> $ PASSPHRASE='i_want+you+to+&duplicate_this' duplicity -v8 --volsize 150\
>     gdocs://test.gdocs.backend.duplicity:address@hidden/tmp\
>    `pwd`/tmp2 
> </shell>    
> 
> Sorry for the long mail...
> Best regards,
> Boris.
> 

saw your mail only just now. sorry. thanks for the detailed explanation! 

i wonder if the correct solution wouldn't be to add mime-types to a list and 
compare against this
 or
abandon testing type alltogether further and simply assuming that anything not 
'folder' is a file.

wrt. --no-compression, yes that is known to be broken since a long time now.

..ede/duply.net