Re: [Duplicity-talk] [Duplicity-team] verify fails on 0.6.18 and 0.7.01

[Rupert Levene]

I can confirm that verify with 0.7.01 works using a local copy of the
backup folder, but fails using the gdocs backup.

The large file is a tiny bit larger than 2^31 bytes:

2147540150 Feb  6 10:58 duplicity-full-signatures.20150205T170035Z.sigtar.gpg

Could an option be added to split signature files into chunks, rather
like --volsize does for the other files?

I don't think the initial backup gave errors, but I'm not certain. If
you like I can try it again, but it does take rather a long time to
run.

Rupert

On 8 February 2015 at 16:32, Kenneth Loafman <address@hidden> wrote:
> Were there any errors during the initial backup?  This looks like a 32-bit
> Python limit.
>
> ...Ken
>
>
>
>
> On Sun, Feb 8, 2015 at 9:16 AM, <address@hidden> wrote:
>>
>> On 08.02.2015 15:46, Rupert Levene wrote:
>> > On 8 February 2015 at 14:32,  <address@hidden> wrote:
>> >> On 08.02.2015 15:05, Rupert Levene wrote:
>> >>> Hi,
>> >>>
>> >>> I have made a large to gdocs using 0.6.18 from Ubuntu 12.04. A 2GB
>> >>> file duplicity-full-signatures.20150205T170035Z.sigtar.gpg was
>> >>> created, and duplicity chokes on this. It pauses for a while (I guess
>> >>> while it downloads the file) and then says
>> >>>
>> >>> Attempt 1 failed: BackendException: Failed to download file
>> >>> 'duplicity-full-signatures.20150205T170035Z.sigtar.gpg' in remote
>> >>> folder 'i7-bigstore': join() result is too long for a Python string
>> >>> Backtrace of previous error: Traceback (innermost last):
>> >>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>> >>> 311, in iterate
>> >>>     return fn(*args, **kwargs)
>> >>>   File
>> >>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>> >>> line 144, in get
>> >>>     % (remote_filename, self.folder.title.text, str(e)), raise_errors)
>> >>>   File
>> >>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>> >>> line 182, in __handle_error
>> >>>     raise BackendException(message)
>> >>>  BackendException: Failed to download file
>> >>> 'duplicity-full-signatures.20150205T170035Z.sigtar.gpg' in remote
>> >>> folder 'i7-bigstore': join() result is too long for a Python string
>> >>>
>> >>> It'll then try again, and give the same error, at which point I hit
>> >>> control-C.
>> >>>
>> >>> Version 0.7.01 fails more quickly:
>> >>>
>> >>> $ apt-cache policy duplicity
>> >>> duplicity:
>> >>>   Installed: 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1
>> >>>   Candidate: 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1
>> >>>   Version table:
>> >>>  *** 0.7.01-0ubuntu0ppa1063~ubuntu12.04.1 0
>> >>>         500 http://ppa.launchpad.net/duplicity-team/ppa/ubuntu/
>> >>> precise/main i386 Packages
>> >>>         100 /var/lib/dpkg/status
>> >>>      0.6.18-0ubuntu3.5 0
>> >>>         500 http://ie.archive.ubuntu.com/ubuntu/ precise-updates/main
>> >>> i386 Packages
>> >>>      0.6.18-0ubuntu3 0
>> >>>         500 http://ie.archive.ubuntu.com/ubuntu/ precise/main i386
>> >>> Packages
>> >>> $ duplicity -V
>> >>> duplicity 0.7.01
>> >>> $ duplicity verify --verbosity '8' --exclude-globbing-filelist
>> >>> /home/rupert/.duply/i7-bigstore/exclude
>> >>> gdocs://address@hidden/backups/duplicity/i7-bigstore
>> >>> /mnt/bigstore
>> >>> Using archive dir: /home/rupert/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxx
>> >>> Using backup name: xxxxxxxxxxxxxxxxxxxxx
>> >>> Import of duplicity.backends.azurebackend Succeeded
>> >>> Import of duplicity.backends.botobackend Succeeded
>> >>> Import of duplicity.backends.cfbackend Succeeded
>> >>> Import of duplicity.backends.copycombackend Succeeded
>> >>> Import of duplicity.backends.dpbxbackend Succeeded
>> >>> Import of duplicity.backends.gdocsbackend Succeeded
>> >>> Import of duplicity.backends.giobackend Succeeded
>> >>> Import of duplicity.backends.hsibackend Succeeded
>> >>> Import of duplicity.backends.hubicbackend Succeeded
>> >>> Import of duplicity.backends.imapbackend Succeeded
>> >>> Import of duplicity.backends.lftpbackend Succeeded
>> >>> Import of duplicity.backends.localbackend Succeeded
>> >>> Import of duplicity.backends.megabackend Succeeded
>> >>> Import of duplicity.backends.ncftpbackend Succeeded
>> >>> Import of duplicity.backends.onedrivebackend Failed: No module named
>> >>> requests
>> >>> Import of duplicity.backends.par2backend Succeeded
>> >>> Import of duplicity.backends.pydrivebackend Succeeded
>> >>> Import of duplicity.backends.rsyncbackend Succeeded
>> >>> Import of duplicity.backends.ssh_paramiko_backend Succeeded
>> >>> Import of duplicity.backends.ssh_pexpect_backend Succeeded
>> >>> Import of duplicity.backends.swiftbackend Succeeded
>> >>> Import of duplicity.backends.sxbackend Succeeded
>> >>> Import of duplicity.backends.tahoebackend Succeeded
>> >>> Import of duplicity.backends.webdavbackend Succeeded
>> >>> Using temporary directory /tmp/duplicity-lCUBfI-tempdir
>> >>> Traceback (most recent call last):
>> >>>   File "/usr/bin/duplicity", line 1497, in <module>
>> >>>     with_tempdir(main)
>> >>>   File "/usr/bin/duplicity", line 1491, in with_tempdir
>> >>>     fn()
>> >>>   File "/usr/bin/duplicity", line 1324, in main
>> >>>     action = commandline.ProcessCommandLine(sys.argv[1:])
>> >>>   File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py",
>> >>> line 1056, in ProcessCommandLine
>> >>>     backup, local_pathname = set_backend(args[0], args[1])
>> >>>   File "/usr/lib/python2.7/dist-packages/duplicity/commandline.py",
>> >>> line 949, in set_backend
>> >>>     globals.backend = backend.get_backend(bend)
>> >>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>> >>> 221, in get_backend
>> >>>     obj = get_backend_object(url_string)
>> >>>   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>> >>> 207, in get_backend_object
>> >>>     return factory(pu)
>> >>>   File
>> >>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>> >>> line 61, in __init__
>> >>>     entries = self._fetch_entries(parent_folder_id, 'folder',
>> >>> folder_name)
>> >>>   File
>> >>> "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>> >>> line 161, in _fetch_entries
>> >>>     entries = self.client.get_all_resources(uri=uri)
>> >>> AttributeError: 'DocsClient' object has no attribute
>> >>> 'get_all_resources'
>> >>>
>> >>> Are these two independent bugs? Is there a workaround or fix?
>> >>>
>> >>
>> >> yes, it looks like two different bugs with two different error
>> >> messages.
>> >>
>> >> first try latest stable duplicity 0.6.25.. there were changes in the
>> >> gdocs backend since your version.
>> >>
>> >> if that fails - can you try downloading your whole backups to a local
>> >> folder and try to restore from there?
>> >>
>> >> ..ede/duply.net
>> >>
>> >> _______________________________________________
>> >> Duplicity-talk mailing list
>> >> address@hidden
>> >> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>> >
>> > I had tried 0.6.25 too, and it was giving the same get_all_resources
>> > error as 0.7.01. However since then I used pip to upgrade gdata, and
>> > 0.7.01 gives me very similar output to my original error from 0.6.18:
>> >
>> > $ duplicity verify --verbosity '9' --exclude-globbing-filelist
>> > /home/rupert/.duply/i7-bigstore/exclude
>> > gdocs://address@hidden/backups/duplicity/i7-bigstore
>> > /mnt/bigstore
>> > Using archive dir:
>> > /home/rupert/.cache/duplicity/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
>> > Using backup name: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
>> > Import of duplicity.backends.azurebackend Succeeded
>> > Import of duplicity.backends.botobackend Succeeded
>> > Import of duplicity.backends.cfbackend Succeeded
>> > Import of duplicity.backends.copycombackend Succeeded
>> > Import of duplicity.backends.dpbxbackend Succeeded
>> > Import of duplicity.backends.gdocsbackend Succeeded
>> > Import of duplicity.backends.giobackend Succeeded
>> > Import of duplicity.backends.hsibackend Succeeded
>> > Import of duplicity.backends.hubicbackend Succeeded
>> > Import of duplicity.backends.imapbackend Succeeded
>> > Import of duplicity.backends.lftpbackend Succeeded
>> > Import of duplicity.backends.localbackend Succeeded
>> > Import of duplicity.backends.megabackend Succeeded
>> > Import of duplicity.backends.ncftpbackend Succeeded
>> > Import of duplicity.backends.onedrivebackend Failed: No module named
>> > requests
>> > Import of duplicity.backends.par2backend Succeeded
>> > Import of duplicity.backends.pydrivebackend Succeeded
>> > Import of duplicity.backends.rsyncbackend Succeeded
>> > Import of duplicity.backends.ssh_paramiko_backend Succeeded
>> > Import of duplicity.backends.ssh_pexpect_backend Succeeded
>> > Import of duplicity.backends.swiftbackend Succeeded
>> > Import of duplicity.backends.sxbackend Succeeded
>> > Import of duplicity.backends.tahoebackend Succeeded
>> > Import of duplicity.backends.webdavbackend Succeeded
>> > Reading globbing filelist /home/rupert/.duply/i7-bigstore/exclude
>> > Main action: verify
>> >
>> > ================================================================================
>> > duplicity 0.7.01 ($reldate)
>> > Args: /usr/bin/duplicity verify --verbosity 9
>> > --exclude-globbing-filelist /home/rupert/.duply/i7-bigstore/exclude
>> > gdocs://address@hidden/backups/duplicity/i7-bigstore
>> > /mnt/bigstore
>> > Linux levene-i7desktop 3.13.0-44-generic #73~precise1-Ubuntu SMP Wed
>> > Dec 17 00:38:38 UTC 2014 i686 i686
>> > /usr/bin/python 2.7.3 (default, Dec 18 2014, 19:03:52)
>> > [GCC 4.6.3]
>> >
>> > ================================================================================
>> > Synchronizing remote metadata to local cache...
>> > Copying duplicity-full-signatures.20150205T170035Z.sigtar.gpg to local
>> > cache.
>> > Using temporary directory /tmp/duplicity-L9vwHt-tempdir
>> > Registering (mktemp) temporary file
>> > /tmp/duplicity-L9vwHt-tempdir/mktemp-_vVr3t-1
>> > Backtrace of previous error: Traceback (innermost last):
>> >   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>> > 361, in inner_retry
>> >     return fn(self, *args)
>> >   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>> > 534, in get
>> >     self.backend._get(remote_filename, local_path)
>> >   File
>> > "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>> > line 110, in _get
>> >     self.client.DownloadResource(entry, local_path.name)
>> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>> > line 369, in download_resource
>> >     self._download_file(uri, file_path, **kwargs)
>> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>> > line 468, in _download_file
>> >     f.write(self._get_content(uri, **kwargs))
>> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>> > line 451, in _get_content
>> >     return server_response.read()
>> >   File "/usr/lib/python2.7/httplib.py", line 541, in read
>> >     return self._read_chunked(amt)
>> >   File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
>> >     return ''.join(value)
>> >  OverflowError: join() result is too long for a Python string
>> >
>> > Attempt 1 failed. OverflowError: join() result is too long for a Python
>> > string
>> > Backtrace of previous error: Traceback (innermost last):
>> >   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>> > 361, in inner_retry
>> >     return fn(self, *args)
>> >   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>> > 534, in get
>> >     self.backend._get(remote_filename, local_path)
>> >   File
>> > "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>> > line 110, in _get
>> >     self.client.DownloadResource(entry, local_path.name)
>> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>> > line 369, in download_resource
>> >     self._download_file(uri, file_path, **kwargs)
>> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>> > line 468, in _download_file
>> >     f.write(self._get_content(uri, **kwargs))
>> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>> > line 451, in _get_content
>> >     return server_response.read()
>> >   File "/usr/lib/python2.7/httplib.py", line 541, in read
>> >     return self._read_chunked(amt)
>> >   File "/usr/lib/python2.7/httplib.py", line 590, in _read_chunked
>> >     value.append(self._safe_read(chunk_left))
>> >   File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read
>> >     chunk = self.fp.read(min(amt, MAXAMOUNT))
>> >   File "/usr/lib/python2.7/socket.py", line 396, in read
>> >     buf.write(data)
>> >  MemoryError: out of memory
>> >
>> > Attempt 2 failed. MemoryError: out of memory
>> > Backtrace of previous error: Traceback (innermost last):
>> >   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>> > 361, in inner_retry
>> >     return fn(self, *args)
>> >   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>> > 534, in get
>> >     self.backend._get(remote_filename, local_path)
>> >   File
>> > "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>> > line 110, in _get
>> >     self.client.DownloadResource(entry, local_path.name)
>> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>> > line 369, in download_resource
>> >     self._download_file(uri, file_path, **kwargs)
>> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>> > line 468, in _download_file
>> >     f.write(self._get_content(uri, **kwargs))
>> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>> > line 451, in _get_content
>> >     return server_response.read()
>> >   File "/usr/lib/python2.7/httplib.py", line 541, in read
>> >     return self._read_chunked(amt)
>> >   File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
>> >     return ''.join(value)
>> >  OverflowError: join() result is too long for a Python string
>> >
>> > Attempt 3 failed. OverflowError: join() result is too long for a Python
>> > string
>> > Backtrace of previous error: Traceback (innermost last):
>> >   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>> > 361, in inner_retry
>> >     return fn(self, *args)
>> >   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>> > 534, in get
>> >     self.backend._get(remote_filename, local_path)
>> >   File
>> > "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>> > line 110, in _get
>> >     self.client.DownloadResource(entry, local_path.name)
>> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>> > line 369, in download_resource
>> >     self._download_file(uri, file_path, **kwargs)
>> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>> > line 468, in _download_file
>> >     f.write(self._get_content(uri, **kwargs))
>> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>> > line 451, in _get_content
>> >     return server_response.read()
>> >   File "/usr/lib/python2.7/httplib.py", line 541, in read
>> >     return self._read_chunked(amt)
>> >   File "/usr/lib/python2.7/httplib.py", line 590, in _read_chunked
>> >     value.append(self._safe_read(chunk_left))
>> >   File "/usr/lib/python2.7/httplib.py", line 647, in _safe_read
>> >     chunk = self.fp.read(min(amt, MAXAMOUNT))
>> >   File "/usr/lib/python2.7/socket.py", line 400, in read
>> >     buf.write(data)
>> >  MemoryError: out of memory
>> >
>> > Attempt 4 failed. MemoryError: out of memory
>> > Backtrace of previous error: Traceback (innermost last):
>> >   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>> > 361, in inner_retry
>> >     return fn(self, *args)
>> >   File "/usr/lib/python2.7/dist-packages/duplicity/backend.py", line
>> > 534, in get
>> >     self.backend._get(remote_filename, local_path)
>> >   File
>> > "/usr/lib/python2.7/dist-packages/duplicity/backends/gdocsbackend.py",
>> > line 110, in _get
>> >     self.client.DownloadResource(entry, local_path.name)
>> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>> > line 369, in download_resource
>> >     self._download_file(uri, file_path, **kwargs)
>> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>> > line 468, in _download_file
>> >     f.write(self._get_content(uri, **kwargs))
>> >   File "/usr/local/lib/python2.7/dist-packages/gdata/docs/client.py",
>> > line 451, in _get_content
>> >     return server_response.read()
>> >   File "/usr/lib/python2.7/httplib.py", line 541, in read
>> >     return self._read_chunked(amt)
>> >   File "/usr/lib/python2.7/httplib.py", line 624, in _read_chunked
>> >     return ''.join(value)
>> >  OverflowError: join() result is too long for a Python string
>> >
>> > Giving up after 5 attempts. OverflowError: join() result is too long
>> > for a Python string
>> > Releasing lockfile <lockfile.LinkFileLock instance at 0x9d0986c>
>> > Removing still remembered temporary file
>> > /tmp/duplicity-L9vwHt-tempdir/mktemp-_vVr3t-1
>> > $
>> >
>> > I am running 32-bit linux with 4gb RAM and 4gb swap.
>> >
>> > I'll try downloading the backup to local storage as you suggest.
>> >
>>
>> please do.. looks like httplib fails on 2GB files and there isn't much we
>> can do about that.
>>
>> Mike, Ken: any idea?
>>
>> ..ede/duply.net
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~duplicity-team
>> Post to     : address@hidden
>> Unsubscribe : https://launchpad.net/~duplicity-team
>> More help   : https://help.launchpad.net/ListHelp
>
>
>
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/duplicity-talk
>