Re: [Duplicity-talk] verify fails on 0.6.18 and 0.7.01

[edgar . soldin]

On 08.02.2015 15:46, Rupert Levene wrote:
> On 8 February 2015 at 14:32,  <address@hidden> wrote:
>> On 08.02.2015 15:05, Rupert Levene wrote:
>>> Hi,
>>>
>>> I have made a large to gdocs using 0.6.18 from Ubuntu 12.04. A 2GB
>>> file duplicity-full-signatures.20150205T170035Z.sigtar.gpg was
>>> created, and duplicity chokes on this. It pauses for a while (I guess
>>> while it downloads the file) and then says
>>>
>>> Attempt 1 failed: BackendException: Failed to download file
>>> 'duplicity-full-signatures.20150205T170035Z.sigtar.gpg' in remote
>>> folder 'i7-bigstore': join() result is too long for a Python string
>>> Backtrace of previous error: Traceback (innermost last):
>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>> 311, in iterate
>>>     return fn(*args, **kwargs)
>>>   File 
>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>> line 144, in get
>>>     % (remote_filename, self.folder.title.text, str(e)), raise_errors)
>>>   File 
>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>> line 182, in __handle_error
>>>     raise BackendException(message)
>>>  BackendException: Failed to download file
>>> 'duplicity-full-signatures.20150205T170035Z.sigtar.gpg' in remote
>>> folder 'i7-bigstore': join() result is too long for a Python string
>>>
>>> It'll then try again, and give the same error, at which point I hit 
>>> control-C.
>>>
>>> Version 0.7.01 fails more quickly:
>>>
>>> $ apt-cache policy duplicity
>>> duplicity:
>>>   Installed: 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1
>>>   Candidate: 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1
>>>   Version table:
>>>  *** 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1 0
>>>         500 http://ppa.launchpad.net/duplicity-team/ppa/ubuntu/
>>> precise/main i386 Packages
>>>         100 /var/lib/dpkg/status
>>>      0.6.18-0ubuntu3.5 0
>>>         500 http://ie.archive.ubuntu.com/ubuntu/ precise-updates/main
>>> i386 Packages
>>>      0.6.18-0ubuntu3 0
>>>         500 http://ie.archive.ubuntu.com/ubuntu/ precise/main i386 Packages
>>> $ duplicity -V
>>> duplicity 0.7.01
>>> $ duplicity verify --verbosity '8' --exclude-globbing-filelist
>>> /home/rupert/.duply/i7-bigstore/exclude
>>> gdocs://address@hidden/backups/duplicity/i7-bigstore
>>> /mnt/bigstore
>>> Using archive dir: /home/rupert/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxx
>>> Using backup name: xxxxxxxxxxxxxxxxxxxxx
>>> Import of duplicity.backends.azurebackend Succeeded
>>> Import of duplicity.backends.botobackend Succeeded
>>> Import of duplicity.backends.cfbackend Succeeded
>>> Import of duplicity.backends.copycombackend Succeeded
>>> Import of duplicity.backends.dpbxbackend Succeeded
>>> Import of duplicity.backends.gdocsbackend Succeeded
>>> Import of duplicity.backends.giobackend Succeeded
>>> Import of duplicity.backends.hsibackend Succeeded
>>> Import of duplicity.backends.hubicbackend Succeeded
>>> Import of duplicity.backends.imapbackend Succeeded
>>> Import of duplicity.backends.lftpbackend Succeeded
>>> Import of duplicity.backends.localbackend Succeeded
>>> Import of duplicity.backends.megabackend Succeeded
>>> Import of duplicity.backends.ncftpbackend Succeeded
>>> Import of duplicity.backends.onedrivebackend Failed: No module named 
>>> requests
>>> Import of duplicity.backends.par2backend Succeeded
>>> Import of duplicity.backends.pydrivebackend Succeeded
>>> Import of duplicity.backends.rsyncbackend Succeeded
>>> Import of duplicity.backends.ssh_paramiko_backend Succeeded
>>> Import of duplicity.backends.ssh_pexpect_backend Succeeded
>>> Import of duplicity.backends.swiftbackend Succeeded
>>> Import of duplicity.backends.sxbackend Succeeded
>>> Import of duplicity.backends.tahoebackend Succeeded
>>> Import of duplicity.backends.webdavbackend Succeeded
>>> Using temporary directory /tmp/duplicity-lCUBfI-tempdir
>>> Traceback (most recent call last):
>>>   File "/usr/bin/duplicity", line 1497, in <module>
>>>     with_tempdir(main)
>>>   File "/usr/bin/duplicity", line 1491, in with_tempdir
>>>     fn()
>>>   File "/usr/bin/duplicity", line 1324, in main
>>>     action = commandline.ProcessCommandLine(sys.argv[1:])
>>>   File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py",
>>> line 1056, in ProcessCommandLine
>>>     backup, local_pathname = set_backend(args[0], args[1])
>>>   File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py",
>>> line 949, in set_backend
>>>     globals.backend = backend.get_backend(bend)
>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>> 221, in get_backend
>>>     obj = get_backend_object(url_string)
>>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>>> 207, in get_backend_object
>>>     return factory(pu)
>>>   File 
>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>> line 61, in __init__
>>>     entries = self._fetch_entries(parent_folder_id, 'folder', folder_name)
>>>   File 
>>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>>> line 161, in _fetch_entries
>>>     entries = self.client.get_all_resources(uri=uri)
>>> AttributeError: 'DocsClient' object has no attribute 'get_all_resources'
>>>
>>> Are these two independent bugs? Is there a workaround or fix?
>>>
>>
>> yes, it looks like two different bugs with two different error messages.
>>
>> first try latest stable duplicity 0.6.25.. there were changes in the gdocs 
>> backend since your version.
>>
>> if that fails - can you try downloading your whole backups to a local folder 
>> and try to restore from there?
>>
>> ..ede/duply.net
>>
>> _______________________________________________
>> Duplicity-talk mailing list
>> address@hidden
>> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
> 
> I had tried 0.6.25 too, and it was giving the same get_all_resources
> error as 0.7.01. However since then I used pip to upgrade gdata, and
> 0.7.01 gives me very similar output to my original error from 0.6.18:
> 
> $ duplicity verify --verbosity '9' --exclude-globbing-filelist
> /home/rupert/.duply/i7-bigstore/exclude
> gdocs://address@hidden/backups/duplicity/i7-bigstore
> /mnt/bigstore
> Using archive dir:
> /home/rupert/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Using backup name: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> Import of duplicity.backends.azurebackend Succeeded
> Import of duplicity.backends.botobackend Succeeded
> Import of duplicity.backends.cfbackend Succeeded
> Import of duplicity.backends.copycombackend Succeeded
> Import of duplicity.backends.dpbxbackend Succeeded
> Import of duplicity.backends.gdocsbackend Succeeded
> Import of duplicity.backends.giobackend Succeeded
> Import of duplicity.backends.hsibackend Succeeded
> Import of duplicity.backends.hubicbackend Succeeded
> Import of duplicity.backends.imapbackend Succeeded
> Import of duplicity.backends.lftpbackend Succeeded
> Import of duplicity.backends.localbackend Succeeded
> Import of duplicity.backends.megabackend Succeeded
> Import of duplicity.backends.ncftpbackend Succeeded
> Import of duplicity.backends.onedrivebackend Failed: No module named requests
> Import of duplicity.backends.par2backend Succeeded
> Import of duplicity.backends.pydrivebackend Succeeded
> Import of duplicity.backends.rsyncbackend Succeeded
> Import of duplicity.backends.ssh_paramiko_backend Succeeded
> Import of duplicity.backends.ssh_pexpect_backend Succeeded
> Import of duplicity.backends.swiftbackend Succeeded
> Import of duplicity.backends.sxbackend Succeeded
> Import of duplicity.backends.tahoebackend Succeeded
> Import of duplicity.backends.webdavbackend Succeeded
> Reading globbing filelist /home/rupert/.duply/i7-bigstore/exclude
> Main action: verify
> ================================================================================
> duplicity 0.7.01 ($reldate)
> Args: /usr/bin/duplicity verify --verbosity 9
> --exclude-globbing-filelist /home/rupert/.duply/i7-bigstore/exclude
> gdocs://address@hidden/backups/duplicity/i7-bigstore
> /mnt/bigstore
> Linux levene-i7desktop 3.13.0-44-generic #73~precise1-Ubuntu SMP Wed
> Dec 17 00:38:38 UTC 2014 i686 i686
> /usr/bin/python 2.7.3 (default, Dec 18 2014, 19:03:52)
> [GCC 4.6.3]
> ================================================================================
> Synchronizing remote metadata to local cache...
> Copying duplicity-full-signatures.20150205T170035Z.sigtar.gpg to local cache.
> Using temporary directory /tmp/duplicity-L9vwHt-tempdir
> Registering (mktemp) temporary file
> /tmp/duplicity-L9vwHt-tempdir/mktemp-_vVr3t-1
> Backtrace of previous error: Traceback (innermost last):
>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
> 361, in inner_retry
>     return fn(self, *args)
>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 534, in 
> get
>     self.backend._get(remote_filename, local_path)
>   File "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
> line 110, in _get
>     self.client.DownloadResource(entry, local_path.name)
>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> line 369, in download_resource
>     self._download_file(uri, file_path, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> line 468, in _download_file
>     f.write(self._get_content(uri, **kwargs))
>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> line 451, in _get_content
>     return server_response.read()
>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>     return self._read_chunked(amt)
>   File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
>     return ''.join(value)
>  OverflowError: join() result is too long for a Python string
> 
> Attempt 1 failed. OverflowError: join() result is too long for a Python string
> Backtrace of previous error: Traceback (innermost last):
>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
> 361, in inner_retry
>     return fn(self, *args)
>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 534, in 
> get
>     self.backend._get(remote_filename, local_path)
>   File "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
> line 110, in _get
>     self.client.DownloadResource(entry, local_path.name)
>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> line 369, in download_resource
>     self._download_file(uri, file_path, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> line 468, in _download_file
>     f.write(self._get_content(uri, **kwargs))
>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> line 451, in _get_content
>     return server_response.read()
>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>     return self._read_chunked(amt)
>   File "/usr/lib/python2.7/httplib.py", line 590, in _read_chunked
>     value.append(self._safe_read(chunk_left))
>   File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read
>     chunk = self.fp.read(min(amt, MAXAMOUNT))
>   File "/usr/lib/python2.7/socket.py", line 396, in read
>     buf.write(data)
>  MemoryError: out of memory
> 
> Attempt 2 failed. MemoryError: out of memory
> Backtrace of previous error: Traceback (innermost last):
>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
> 361, in inner_retry
>     return fn(self, *args)
>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 534, in 
> get
>     self.backend._get(remote_filename, local_path)
>   File "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
> line 110, in _get
>     self.client.DownloadResource(entry, local_path.name)
>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> line 369, in download_resource
>     self._download_file(uri, file_path, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> line 468, in _download_file
>     f.write(self._get_content(uri, **kwargs))
>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> line 451, in _get_content
>     return server_response.read()
>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>     return self._read_chunked(amt)
>   File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
>     return ''.join(value)
>  OverflowError: join() result is too long for a Python string
> 
> Attempt 3 failed. OverflowError: join() result is too long for a Python string
> Backtrace of previous error: Traceback (innermost last):
>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
> 361, in inner_retry
>     return fn(self, *args)
>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 534, in 
> get
>     self.backend._get(remote_filename, local_path)
>   File "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
> line 110, in _get
>     self.client.DownloadResource(entry, local_path.name)
>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> line 369, in download_resource
>     self._download_file(uri, file_path, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> line 468, in _download_file
>     f.write(self._get_content(uri, **kwargs))
>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> line 451, in _get_content
>     return server_response.read()
>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>     return self._read_chunked(amt)
>   File "/usr/lib/python2.7/httplib.py", line 590, in _read_chunked
>     value.append(self._safe_read(chunk_left))
>   File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read
>     chunk = self.fp.read(min(amt, MAXAMOUNT))
>   File "/usr/lib/python2.7/socket.py", line 400, in read
>     buf.write(data)
>  MemoryError: out of memory
> 
> Attempt 4 failed. MemoryError: out of memory
> Backtrace of previous error: Traceback (innermost last):
>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
> 361, in inner_retry
>     return fn(self, *args)
>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line 534, in 
> get
>     self.backend._get(remote_filename, local_path)
>   File "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
> line 110, in _get
>     self.client.DownloadResource(entry, local_path.name)
>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> line 369, in download_resource
>     self._download_file(uri, file_path, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> line 468, in _download_file
>     f.write(self._get_content(uri, **kwargs))
>   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
> line 451, in _get_content
>     return server_response.read()
>   File "/usr/lib/python2.7/httplib.py", line 541, in read
>     return self._read_chunked(amt)
>   File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
>     return ''.join(value)
>  OverflowError: join() result is too long for a Python string
> 
> Giving up after 5 attempts. OverflowError: join() result is too long
> for a Python string
> Releasing lockfile <lockfile.LinkFileLock instance at 0x9d0986c>
> Removing still remembered temporary file
> /tmp/duplicity-L9vwHt-tempdir/mktemp-_vVr3t-1
> $
> 
> I am running 32-bit linux with 4gb RAM and 4gb swap.
> 
> I'll try downloading the backup to local storage as you suggest.
> 

please do.. looks like httplib fails on 2GB files and there isn't much we can 
do about that.

Mike, Ken: any idea?

..ede/duply.net