[Duplicity-talk] Weird error message after incremental backup of large drive with a bunch of changed files

[Jakob Bohm]

Dear group,

I have set up some scripts to do various parts of a full system backup
via duplicity to a geographically close S3 bucket (AWS Stockholm),
however for the largest drive, I occasionally experience hangs/errors
near the end of each backup, with the progress display wobbling between
"stalled" and less than 30 minutes left (this time I observed as low as
16 minutes at one point).

Then a day into the stall/short time phase, I received to following
error message (redacted bucket name, mountpoints etc.):

Attempt of put Nr. 1 failed. S3UploadFailedError: Failed to upload
/duplicitypart/tmp/commonprefix-partname/duplicity-g3cvly63-tempdir/duplicity_temp.1/commonprefix-partname_duplicity-inc.20230212T043018Z.to.20230311T203857Z.vol1501.difftar.gpg.vol000+200.par2 
to 
bucketname/partname/commonprefix-partname_duplicity-inc.20230212T043018Z.to.20230311T203857Z.vol1501.difftar.gpg.vol000+200.par2: 
An error occurred (RequestTimeTooSkewed) when calling the 
CreateMultipartUpload operation: The difference between the request time 
and the current time is too large.

After the message there were some small increases in the GB counter in 
the progress bar

ls report after killing duplicity:

-rw-r--r-- 1 root root 21336776 Mar 13 18:33 
/duplicitypart/tmp/commonprefix-partname/duplicity-g3cvly63-tempdir/duplicity_temp.1/commonprefix-partname_duplicity-inc.20230212T043018Z.to.20230311T203857Z.vol1501.difftar.gpg.vol000+200.par2


Currently invoking duplicity 1.2.1 (patched) with command line

duplicity incremental --name commonprefix-partname \
  --archive-dir /duplicitypart/archive/commonprefix-partname \
  --asynchronous-upload \
  --file-prefix commonprefix-partname_ \
  --tempdir /duplicitypart/tmp/commonprefix-partname \
  --verbosity notice \
  --progress \
  --log-file 
/duplicitypart/tmp/commonprefix-partname/log_20230311T20_38_52.log \
  --gpg-options '--homedir /configdir/.gnupg --compress-algo=bzip2' \
  --encrypt-secret-keyring /configdir/.gnupg/secret.gpg \
  --encrypt-key 1234567890ABCDEF1234567890ABCDEF12345678 \
  --sign-key 1234567890ABCDEF1234567890ABCDEF12345678 \
  --hidden-encrypt-key 1234567890ABCDEF1234567890ABCDEF12345678 \
  --exclude-other-filesystems \
  --full-if-older-than 3M \
  --s3-use-multiprocessing \
  --numeric-owner \
  /partname \
  par2+boto3+s3://bucketname/partname

OS: Debian GNU/Linux 11.7 (bullseye) with Python 3.9.2, python-boto3 
version 1.13.14-1

I suspect a relationship with issue #254, and as you see, I have 
incorporated some of the workarounds into the command line.

What are the appropriate troubleshooting steps?


Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded