Re: [Duplicity-talk] [Duplicity-team] verify fails on 0.6.18 and 0.7.01

[edgar . soldin]

good to hear. if you want you may find out where gdocs python API people hang 
out and make them aware of the issue.

we have this url in our manpage
 https://code.google.com/p/gdata-python-client/

..ede/duply.net

On 09.02.2015 17:05, Rupert Levene wrote:
> I can confirm that verify with 0.7.01 works using a local copy of the
> backup folder, but fails using the gdocs backup.
> 
> The large file is a tiny bit larger than 2^31 bytes:
> 
> 2147540150 Feb  6 10:58 duplicity-full-signatures.20150205T170035Z.sigtar.gpg
> 
> Could an option be added to split signature files into chunks, rather
> like --volsize does for the other files?
> 
> I don't think the initial backup gave errors, but I'm not certain. If
> you like I can try it again, but it does take rather a long time to
> run.
> 
> Rupert
> 
> On 8 February 2015 at 16:32, Kenneth Loafman <address@hidden> wrote:
>> Were there any errors during the initial backup?  This looks like a 32-bit
>> Python limit.
>>
>> ...Ken
>>
>>
>>
>>
>> On Sun, Feb 8, 2015 at 9:16 AM, <address@hidden> wrote:
>>>
>>> On 08.02.2015 15:46, Rupert Levene wrote:
>>>> On 8 February 2015 at 14:32,  <address@hidden> wrote:
>>>>> On 08.02.2015 15:05, Rupert Levene wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I have made a large to gdocs using 0.6.18 from Ubuntu 12.04. A 2GB
>>>>>> file duplicity-full-signatures.20150205T170035Z.sigtar.gpg was
>>>>>> created, and duplicity chokes on this. It pauses for a while (I guess
>>>>>> while it downloads the file) and then says
>>>>>>
>>>>>> Attempt 1 failed: BackendException: Failed to download file
>>>>>> 'duplicity-full-signatures.20150205T170035Z.sigtar.gpg' in remote
>>>>>> folder 'i7-bigstore': join() result is too long for a Python string
>>>>>> Backtrace of previous error: Traceback (innermost last):
>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>>> 311, in iterate
>>>>>>     return fn(*args, **kwargs)
>>>>>>   File
>>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>>> line 144, in get
>>>>>>     % (remote_filename, self.folder.title.text, str(e)), raise_errors)
>>>>>>   File
>>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>>> line 182, in __handle_error
>>>>>>     raise BackendException(message)
>>>>>>  BackendException: Failed to download file
>>>>>> 'duplicity-full-signatures.20150205T170035Z.sigtar.gpg' in remote
>>>>>> folder 'i7-bigstore': join() result is too long for a Python string
>>>>>>
>>>>>> It'll then try again, and give the same error, at which point I hit
>>>>>> control-C.
>>>>>>
>>>>>> Version 0.7.01 fails more quickly:
>>>>>>
>>>>>> $ apt-cache policy duplicity
>>>>>> duplicity:
>>>>>>   Installed: 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1
>>>>>>   Candidate: 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1
>>>>>>   Version table:
>>>>>>  *** 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1 0
>>>>>>         500 http://ppa.launchpad.net/duplicity-team/ppa/ubuntu/
>>>>>> precise/main i386 Packages
>>>>>>         100 /var/lib/dpkg/status
>>>>>>      0.6.18-0ubuntu3.5 0
>>>>>>         500 http://ie.archive.ubuntu.com/ubuntu/ precise-updates/main
>>>>>> i386 Packages
>>>>>>      0.6.18-0ubuntu3 0
>>>>>>         500 http://ie.archive.ubuntu.com/ubuntu/ precise/main i386
>>>>>> Packages
>>>>>> $ duplicity -V
>>>>>> duplicity 0.7.01
>>>>>> $ duplicity verify --verbosity '8' --exclude-globbing-filelist
>>>>>> /home/rupert/.duply/i7-bigstore/exclude
>>>>>> gdocs://address@hidden/backups/duplicity/i7-bigstore
>>>>>> /mnt/bigstore
>>>>>> Using archive dir: /home/rupert/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxx
>>>>>> Using backup name: xxxxxxxxxxxxxxxxxxxxx
>>>>>> Import of duplicity.backends.azurebackend Succeeded
>>>>>> Import of duplicity.backends.botobackend Succeeded
>>>>>> Import of duplicity.backends.cfbackend Succeeded
>>>>>> Import of duplicity.backends.copycombackend Succeeded
>>>>>> Import of duplicity.backends.dpbxbackend Succeeded
>>>>>> Import of duplicity.backends.gdocsbackend Succeeded
>>>>>> Import of duplicity.backends.giobackend Succeeded
>>>>>> Import of duplicity.backends.hsibackend Succeeded
>>>>>> Import of duplicity.backends.hubicbackend Succeeded
>>>>>> Import of duplicity.backends.imapbackend Succeeded
>>>>>> Import of duplicity.backends.lftpbackend Succeeded
>>>>>> Import of duplicity.backends.localbackend Succeeded
>>>>>> Import of duplicity.backends.megabackend Succeeded
>>>>>> Import of duplicity.backends.ncftpbackend Succeeded
>>>>>> Import of duplicity.backends.onedrivebackend Failed: No module named
>>>>>> requests
>>>>>> Import of duplicity.backends.par2backend Succeeded
>>>>>> Import of duplicity.backends.pydrivebackend Succeeded
>>>>>> Import of duplicity.backends.rsyncbackend Succeeded
>>>>>> Import of duplicity.backends.ssh_paramiko_backend Succeeded
>>>>>> Import of duplicity.backends.ssh_pexpect_backend Succeeded
>>>>>> Import of duplicity.backends.swiftbackend Succeeded
>>>>>> Import of duplicity.backends.sxbackend Succeeded
>>>>>> Import of duplicity.backends.tahoebackend Succeeded
>>>>>> Import of duplicity.backends.webdavbackend Succeeded
>>>>>> Using temporary directory /tmp/duplicity-lCUBfI-tempdir
>>>>>> Traceback (most recent call last):
>>>>>>   File "/usr/bin/duplicity", line 1497, in <module>
>>>>>>     with_tempdir(main)
>>>>>>   File "/usr/bin/duplicity", line 1491, in with_tempdir
>>>>>>     fn()
>>>>>>   File "/usr/bin/duplicity", line 1324, in main
>>>>>>     action = commandline.ProcessCommandLine(sys.argv[1:])
>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py",
>>>>>> line 1056, in ProcessCommandLine
>>>>>>     backup, local_pathname = set_backend(args[0], args[1])
>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py",
>>>>>> line 949, in set_backend
>>>>>>     globals.backend = backend.get_backend(bend)
>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>>> 221, in get_backend
>>>>>>     obj = get_backend_object(url_string)
>>>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>>>> 207, in get_backend_object
>>>>>>     return factory(pu)
>>>>>>   File
>>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>>> line 61, in __init__
>>>>>>     entries = self._fetch_entries(parent_folder_id, 'folder',
>>>>>> folder_name)
>>>>>>   File
>>>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>>>> line 161, in _fetch_entries
>>>>>>     entries = self.client.get_all_resources(uri=uri)
>>>>>> AttributeError: 'DocsClient' object has no attribute
>>>>>> 'get_all_resources'
>>>>>>
>>>>>> Are these two independent bugs? Is there a workaround or fix?
>>>>>>
>>>>>
>>>>> yes, it looks like two different bugs with two different error
>>>>> messages.
>>>>>
>>>>> first try latest stable duplicity 0.6.25.. there were changes in the
>>>>> gdocs backend since your version.
>>>>>
>>>>> if that fails - can you try downloading your whole backups to a local
>>>>> folder and try to restore from there?
>>>>>
>>>>> ..ede/duply.net
>>>>>
>>>>> _______________________________________________
>>>>> Duplicity-talk mailing list
>>>>> address@hidden
>>>>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>>>
>>>> I had tried 0.6.25 too, and it was giving the same get_all_resources
>>>> error as 0.7.01. However since then I used pip to upgrade gdata, and
>>>> 0.7.01 gives me very similar output to my original error from 0.6.18:
>>>>
>>>> $ duplicity verify --verbosity '9' --exclude-globbing-filelist
>>>> /home/rupert/.duply/i7-bigstore/exclude
>>>> gdocs://address@hidden/backups/duplicity/i7-bigstore
>>>> /mnt/bigstore
>>>> Using archive dir:
>>>> /home/rupert/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
>>>> Using backup name: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
>>>> Import of duplicity.backends.azurebackend Succeeded
>>>> Import of duplicity.backends.botobackend Succeeded
>>>> Import of duplicity.backends.cfbackend Succeeded
>>>> Import of duplicity.backends.copycombackend Succeeded
>>>> Import of duplicity.backends.dpbxbackend Succeeded
>>>> Import of duplicity.backends.gdocsbackend Succeeded
>>>> Import of duplicity.backends.giobackend Succeeded
>>>> Import of duplicity.backends.hsibackend Succeeded
>>>> Import of duplicity.backends.hubicbackend Succeeded
>>>> Import of duplicity.backends.imapbackend Succeeded
>>>> Import of duplicity.backends.lftpbackend Succeeded
>>>> Import of duplicity.backends.localbackend Succeeded
>>>> Import of duplicity.backends.megabackend Succeeded
>>>> Import of duplicity.backends.ncftpbackend Succeeded
>>>> Import of duplicity.backends.onedrivebackend Failed: No module named
>>>> requests
>>>> Import of duplicity.backends.par2backend Succeeded
>>>> Import of duplicity.backends.pydrivebackend Succeeded
>>>> Import of duplicity.backends.rsyncbackend Succeeded
>>>> Import of duplicity.backends.ssh_paramiko_backend Succeeded
>>>> Import of duplicity.backends.ssh_pexpect_backend Succeeded
>>>> Import of duplicity.backends.swiftbackend Succeeded
>>>> Import of duplicity.backends.sxbackend Succeeded
>>>> Import of duplicity.backends.tahoebackend Succeeded
>>>> Import of duplicity.backends.webdavbackend Succeeded
>>>> Reading globbing filelist /home/rupert/.duply/i7-bigstore/exclude
>>>> Main action: verify
>>>>
>>>> ================================================================================
>>>> duplicity 0.7.01 ($reldate)
>>>> Args: /usr/bin/duplicity verify --verbosity 9
>>>> --exclude-globbing-filelist /home/rupert/.duply/i7-bigstore/exclude
>>>> gdocs://address@hidden/backups/duplicity/i7-bigstore
>>>> /mnt/bigstore
>>>> Linux levene-i7desktop 3.13.0-44-generic #73~precise1-Ubuntu SMP Wed
>>>> Dec 17 00:38:38 UTC 2014 i686 i686
>>>> /usr/bin/python 2.7.3 (default, Dec 18 2014, 19:03:52)
>>>> [GCC 4.6.3]
>>>>
>>>> ================================================================================
>>>> Synchronizing remote metadata to local cache...
>>>> Copying duplicity-full-signatures.20150205T170035Z.sigtar.gpg to local
>>>> cache.
>>>> Using temporary directory /tmp/duplicity-L9vwHt-tempdir
>>>> Registering (mktemp) temporary file
>>>> /tmp/duplicity-L9vwHt-tempdir/mktemp-_vVr3t-1
>>>> Backtrace of previous error: Traceback (innermost last):
>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>> 361, in inner_retry
>>>>     return fn(self, *args)
>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>> 534, in get
>>>>     self.backend._get(remote_filename, local_path)
>>>>   File
>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>> line 110, in _get
>>>>     self.client.DownloadResource(entry, local_path.name)
>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>> line 369, in download_resource
>>>>     self._download_file(uri, file_path, **kwargs)
>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>> line 468, in _download_file
>>>>     f.write(self._get_content(uri, **kwargs))
>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>> line 451, in _get_content
>>>>     return server_response.read()
>>>>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>>>>     return self._read_chunked(amt)
>>>>   File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
>>>>     return ''.join(value)
>>>>  OverflowError: join() result is too long for a Python string
>>>>
>>>> Attempt 1 failed. OverflowError: join() result is too long for a Python
>>>> string
>>>> Backtrace of previous error: Traceback (innermost last):
>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>> 361, in inner_retry
>>>>     return fn(self, *args)
>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>> 534, in get
>>>>     self.backend._get(remote_filename, local_path)
>>>>   File
>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>> line 110, in _get
>>>>     self.client.DownloadResource(entry, local_path.name)
>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>> line 369, in download_resource
>>>>     self._download_file(uri, file_path, **kwargs)
>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>> line 468, in _download_file
>>>>     f.write(self._get_content(uri, **kwargs))
>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>> line 451, in _get_content
>>>>     return server_response.read()
>>>>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>>>>     return self._read_chunked(amt)
>>>>   File "/usr/lib/python2.7/httplib.py", line 590, in _read_chunked
>>>>     value.append(self._safe_read(chunk_left))
>>>>   File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read
>>>>     chunk = self.fp.read(min(amt, MAXAMOUNT))
>>>>   File "/usr/lib/python2.7/socket.py", line 396, in read
>>>>     buf.write(data)
>>>>  MemoryError: out of memory
>>>>
>>>> Attempt 2 failed. MemoryError: out of memory
>>>> Backtrace of previous error: Traceback (innermost last):
>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>> 361, in inner_retry
>>>>     return fn(self, *args)
>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>> 534, in get
>>>>     self.backend._get(remote_filename, local_path)
>>>>   File
>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>> line 110, in _get
>>>>     self.client.DownloadResource(entry, local_path.name)
>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>> line 369, in download_resource
>>>>     self._download_file(uri, file_path, **kwargs)
>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>> line 468, in _download_file
>>>>     f.write(self._get_content(uri, **kwargs))
>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>> line 451, in _get_content
>>>>     return server_response.read()
>>>>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>>>>     return self._read_chunked(amt)
>>>>   File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
>>>>     return ''.join(value)
>>>>  OverflowError: join() result is too long for a Python string
>>>>
>>>> Attempt 3 failed. OverflowError: join() result is too long for a Python
>>>> string
>>>> Backtrace of previous error: Traceback (innermost last):
>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>> 361, in inner_retry
>>>>     return fn(self, *args)
>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>> 534, in get
>>>>     self.backend._get(remote_filename, local_path)
>>>>   File
>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>> line 110, in _get
>>>>     self.client.DownloadResource(entry, local_path.name)
>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>> line 369, in download_resource
>>>>     self._download_file(uri, file_path, **kwargs)
>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>> line 468, in _download_file
>>>>     f.write(self._get_content(uri, **kwargs))
>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>> line 451, in _get_content
>>>>     return server_response.read()
>>>>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>>>>     return self._read_chunked(amt)
>>>>   File "/usr/lib/python2.7/httplib.py", line 590, in _read_chunked
>>>>     value.append(self._safe_read(chunk_left))
>>>>   File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read
>>>>     chunk = self.fp.read(min(amt, MAXAMOUNT))
>>>>   File "/usr/lib/python2.7/socket.py", line 400, in read
>>>>     buf.write(data)
>>>>  MemoryError: out of memory
>>>>
>>>> Attempt 4 failed. MemoryError: out of memory
>>>> Backtrace of previous error: Traceback (innermost last):
>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>> 361, in inner_retry
>>>>     return fn(self, *args)
>>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>>> 534, in get
>>>>     self.backend._get(remote_filename, local_path)
>>>>   File
>>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>>> line 110, in _get
>>>>     self.client.DownloadResource(entry, local_path.name)
>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>> line 369, in download_resource
>>>>     self._download_file(uri, file_path, **kwargs)
>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>> line 468, in _download_file
>>>>     f.write(self._get_content(uri, **kwargs))
>>>>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>>>> line 451, in _get_content
>>>>     return server_response.read()
>>>>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>>>>     return self._read_chunked(amt)
>>>>   File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
>>>>     return ''.join(value)
>>>>  OverflowError: join() result is too long for a Python string
>>>>
>>>> Giving up after 5 attempts. OverflowError: join() result is too long
>>>> for a Python string
>>>> Releasing lockfile <lockfile.LinkFileLock instance at 0x9d0986c>
>>>> Removing still remembered temporary file
>>>> /tmp/duplicity-L9vwHt-tempdir/mktemp-_vVr3t-1
>>>> $
>>>>
>>>> I am running 32-bit linux with 4gb RAM and 4gb swap.
>>>>
>>>> I'll try downloading the backup to local storage as you suggest.
>>>>
>>>
>>> please do.. looks like httplib fails on 2GB files and there isn't much we
>>> can do about that.
>>>
>>> Mike, Ken: any idea?
>>>
>>> ..ede/duply.net
>>>
>>> _______________________________________________
>>> Mailing list: https://launchpad.net/~duplicity-team
>>> Post to     : address@hidden
>>> Unsubscribe : https://launchpad.net/~duplicity-team
>>> More help   : https://help.launchpad.net/ListHelp
>>
>>
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>>