Re: [Duplicity-talk] Unexpected end of data (2)

[S. Breedveld]

Ok, this could indeed be it. The file reads:

2147402630 Sep  2 15:08 duplicity-full-signatures.20160727T201540Z.sigtar.gz

Which is pretty close to 2 GiB exactly. The file on the server (which is a .gpg btw), is slightly larger (2.1 GiB).

The filesystem can handle large files (ext4+ecryptfs) but the CPU is 32 bit, so the bottleneck is there is probably in one of the commands to decrypt the sigtar file. Questions:

1) Is there a simple way for duplicity to split the tarfile?

Probably not, so:

2) How does this file "grow"? Solely by number of files? In that case, is it a solution to split the backup in several separate, smaller runs?

Thanks for the quick response!

On Mon, 2016-09-05 at 11:17 -0500, Kenneth Loafman wrote:

Usually this means that the input tarfile has been truncated.

This could be the sigtar file in this case.  Is it possible that Synology has an upper limit on file size and is silently truncating the signature file?  I'm not familiar with your environment, so we're going to have to guess for a bit.

If you saved the log file, please paste it to pastebin.com and reply with the link they supply.

On Sun, Sep 4, 2016 at 4:13 PM, S. Breedveld via Duplicity-talk <address@hidden> wrote:

Dear all,

I want to append to this previous thread, raised by Adam M.:

https://lists.nongnu.org/archive/html/duplicity-talk/2016-07/msg00002.html

I am coping with the exact same problem. I initially followed the suggestion to remove the latest incremental backups, but this did not solve the problem. Now I have done a new, full backup, which completed well, but the problem reappears with the first incremental step.

I am running duplicity on a Synology (DS111, running armv5tel CPU), based on the build by SynoCommunity ( https://synocommunity.com/ ). To keep things as simple as possible, I initiated a "local" backup to a remote filesystem, mounted locally (sshfs). 

Of course I checked the basics, i.e. free disk space, permissions, etc., and does not seem to be the issue.

Command line:

/usr/local/bin/duplicity --exclude-regexp '.*[Nn][Oo][Bb][Aa][Cc][Kk][Uu][Pp].*' --archive-dir=/volume1/homes/sebastiaan/duplicity/ --tempdir=/volume1/homes/sebastiaan/duplicity/ --verbosity 9 --asynchronous-upload /volume1/prive/ file:///volume1/homes/sebastiaan/remotemountpoint/

Start output:

Using archive dir: /volume1/homes/sebastiaan/duplicity/7fd35f7d883ae08eaaf74c8830badf29

Using backup name: 7fd35f7d883ae08eaaf74c8830badf29

Import of duplicity.backends.azurebackend Succeeded

Import of duplicity.backends.botobackend Succeeded

Import of duplicity.backends.cfbackend Succeeded

Import of duplicity.backends.copycombackend Succeeded

Import of duplicity.backends.dpbxbackend Succeeded

Import of duplicity.backends.gdocsbackend Succeeded

Import of duplicity.backends.giobackend Succeeded

Import of duplicity.backends.hsibackend Succeeded

Import of duplicity.backends.hubicbackend Succeeded

Import of duplicity.backends.imapbackend Succeeded

Import of duplicity.backends.lftpbackend Succeeded

Import of duplicity.backends.localbackend Succeeded

Import of duplicity.backends.megabackend Succeeded

Import of duplicity.backends.multibackend Succeeded

Import of duplicity.backends.ncftpbackend Succeeded

Import of duplicity.backends.onedrivebackend Succeeded

Import of duplicity.backends.par2backend Succeeded

Import of duplicity.backends.pydrivebackend Succeeded

Import of duplicity.backends.rsyncbackend Succeeded

Import of duplicity.backends.ssh_paramiko_backend Succeeded

Import of duplicity.backends.ssh_pexpect_backend Succeeded

Import of duplicity.backends.swiftbackend Succeeded

Import of duplicity.backends.sxbackend Succeeded

Import of duplicity.backends.tahoebackend Succeeded

Import of duplicity.backends.webdavbackend Succeeded

Main action: inc

================================================================================

duplicity 0.7.05 (September 15, 2015)

Args: /usr/local/bin/duplicity --exclude-regexp '.*[Nn][Oo][Bb][Aa][Cc][Kk][Uu][Pp].*' --archive-dir=/volume1/homes/sebastiaan/duplicity/ --tempdir=/volume1/homes/sebastiaan/duplicity/ --verbosity 9 --asynchronous-upload /volume1/prive/ file:///volume1/homes/sebastiaan/remotemountpoint

Linux HomeStore 2.6.32.12 #7393 Thu Jun 2 19:48:48 CST 2016 armv5tel 

/usr/local/duplicity/env/bin/python 2.7.11 (default, Dec 17 2015, 11:15:51) 

[GCC 4.6.4]

================================================================================

Using temporary directory /volume1/homes/sebastiaan/duplicity/duplicity-08NDGT-tempdir

Registering (mkstemp) temporary file /volume1/homes/sebastiaan/duplicity/duplicity-08NDGT-tempdir/mkstemp-ynGqb_-1

Temp has 344093282304 available, backup will use approx 60293120.

Local and Remote metadata are synchronized, no sync needed.

13117 files exist on backend

10 files exist in cache

Extracting backup chains from list of files: [u'duplicity-full.20160727T201540Z.vol1690.difftar.gpg'....

Later informative part:

File duplicity-full.20160727T201540Z.vol11895.difftar.gpg is part of known set

Found backup chain [Wed Jul 27 22:15:40 2016]-[Wed Jul 27 22:15:40 2016]

Last full backup date: Wed Jul 27 22:15:40 2016

Collection Status

-----------------

Connecting with backend: BackendWrapper

Archive dir: /volume1/homes/sebastiaan/duplicity/7fd35f7d883ae08eaaf74c8830badf29

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:

-------------------------

Chain start time: Wed Jul 27 22:15:40 2016

Chain end time: Wed Jul 27 22:15:40 2016

Number of contained backup sets: 1

Total number of contained volumes: 13115

 Type of backup set:                            Time:      Num volumes:

                Full         Wed Jul 27 22:15:40 2016             13115

-------------------------

No orphaned or incomplete backup sets found.

Registering (mktemp) temporary file /volume1/homes/sebastiaan/duplicity/duplicity-08NDGT-tempdir/mktemp-5ZMA51-2

Deleting /volume1/homes/sebastiaan/duplicity/duplicity-08NDGT-tempdir/mktemp-5ZMA51-2

Forgetting temporary file /volume1/homes/sebastiaan/duplicity/duplicity-08NDGT-tempdir/mktemp-5ZMA51-2

Using temporary directory /volume1/homes/sebastiaan/duplicity/7fd35f7d883ae08eaaf74c8830badf29/duplicity-Xz3QdP-tempdir

Registering (mktemp) temporary file /volume1/homes/sebastiaan/duplicity/7fd35f7d883ae08eaaf74c8830badf29/duplicity-Xz3QdP-tempdir/mktemp-dpdZXg-1

Using temporary directory /volume1/homes/sebastiaan/duplicity/7fd35f7d883ae08eaaf74c8830badf29/duplicity-O4M9YM-tempdir

Registering (mktemp) temporary file /volume1/homes/sebastiaan/duplicity/7fd35f7d883ae08eaaf74c8830badf29/duplicity-O4M9YM-tempdir/mktemp-_Hw4en-1

AsyncScheduler: instantiating at concurrency 1

Registering (mktemp) temporary file /volume1/homes/sebastiaan/duplicity/duplicity-08NDGT-tempdir/mktemp-jED6Ib-3

Selecting /volume1/prive

Comparing . and .

Selection: examining path /volume1/prive/#recycle

Selection:     result: None from function: Regular _expression_: '.*[Nn][Oo][Bb][Aa][Cc][Kk][Uu][Pp].*'

Selection:     + including file

Selecting /volume1/prive/#recycle

...

And then:

Selecting /volume1/prive/sebastiaan/Video/07/VIDEO_TS/VTS_01_3.VOB

Releasing lockfile <LinkLockFile: '/volume1/homes/sebastiaan/duplicity/7fd35f7d883ae08eaaf74c8830badf29/HomeStore.20136-1083199126' -- '/volume1/homes/sebastiaan/duplicity/7fd35f7d883ae08eaaf74c8830badf29/lockfile'>

Removing still remembered temporary file /volume1/homes/sebastiaan/duplicity/duplicity-2iR3gQ-tempdir/mktemp-tGnUdl-3

Removing still remembered temporary file /volume1/homes/sebastiaan/duplicity/duplicity-2iR3gQ-tempdir/mkstemp-Ny13R0-1

Traceback (most recent call last):

  File "/usr/local/bin/duplicity", line 1525, in <module>

    with_tempdir(main)

  File "/usr/local/bin/duplicity", line 1519, in with_tempdir

    fn()

  File "/usr/local/bin/duplicity", line 1373, in main

    do_backup(action)

  File "/usr/local/bin/duplicity", line 1501, in do_backup

    incremental_backup(sig_chain)

  File "/usr/local/bin/duplicity", line 658, in incremental_backup

    globals.backend)

  File "/usr/local/bin/duplicity", line 423, in write_multivol

    globals.gpg_profile, globals.volsize)

  File "/usr/local/duplicity/env/lib/python2.7/site-packages/duplicity/gpg.py", line 340, in GPGWriteFile

    data = "">
  File "/usr/local/duplicity/env/lib/python2.7/site-packages/duplicity/diffdir.py", line 523, in next
    result = self.process(self.input_iter.next())
  File "/usr/local/duplicity/env/lib/python2.7/site-packages/duplicity/diffdir.py", line 195, in get_delta_iter
    for new_path, sig_path in collated:
  File "/usr/local/duplicity/env/lib/python2.7/site-packages/duplicity/diffdir.py", line 286, in collate2iters
    relem2 = riter2.next()
  File "/usr/local/duplicity/env/lib/python2.7/site-packages/duplicity/diffdir.py", line 354, in combine_path_iters
    refresh_triple_list(triple_list)
  File "/usr/local/duplicity/env/lib/python2.7/site-packages/duplicity/diffdir.py", line 341, in refresh_triple_list
    new_triple = get_triple(old_triple[1])
  File "/usr/local/duplicity/env/lib/python2.7/site-packages/duplicity/diffdir.py", line 327, in get_triple
    path = path_iter_list[iter_index].next()
  File "/usr/local/duplicity/env/lib/python2.7/site-packages/duplicity/diffdir.py", line 239, in sigtar2path_iter
    for tarinfo in tf:
  File "/usr/local/python/lib/python2.7/tarfile.py", line 2507, in next
    tarinfo = self.tarfile.next()
  File "/usr/local/python/lib/python2.7/tarfile.py", line 2349, in next
    raise ReadError("unexpected end of data")
ReadError: unexpected end of data


Removing still remembered temporary file /volume1/homes/sebastiaan/duplicity/7fd35f7d883ae08eaaf74c8830badf29/duplicity-JIQdoh-tempdir/mktemp-SRkW8S-1
Removing still remembered temporary file /volume1/homes/sebastiaan/duplicity/7fd35f7d883ae08eaaf74c8830badf29/duplicity-w85sue-tempdir/mktemp-WxgULW-1





I have done a few runs, and sometimes it hangs at this file, sometimes the next one...



Any hints here?



Sincerely,
Sebastiaan

_______________________________________________
Duplicity-talk mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

--

Dr.ir. Sebastiaan Breedveld (PhD)
Scientific Researcher

Erasmus MC - Cancer Institute
Department of Radiation Oncology
P.O. Box 5201, 3008 AE Rotterdam, The Netherlands
Visiting address: Groene Hilledijk 301, 3075 EA Rotterdam, The Netherlands, room Gs-20
E address@hidden | T +31 10 70 42 693
www.erasmusmc.nl/kankerinstituut
http://sebastiaanbreedveld.nl/