[Duplicity-talk] SHA1 hash mismatch (I see previous messages as old as 2003)

[M. Ini]

I would really like to be able to use duplicity, but apparently there is a problem. Any time, some SHA1 hash mismatch error can appear, even when verifying a backup right after having created it.

I'm storing the backup files into an external USB hard disk which is not giving any problems even with tar archives of various GB each.

I'm on Ubuntu 18.04, updated almost daily, I installed the latest stable package for duplicity version 0.7.18.2 from this PPA: https://launchpad.net/~duplicity-team/+archive/ubuntu/ppa
(I had tried at first the version currently available through the default repos configured in Ubuntu 18.04, duplicity 0.7.17.x)

I was at first suspecting a gpg/libcrypt problem due to the fact that I was using keys created on another OS with different GPG/libcrypt versions, so I tried with keys created on Ubuntu 18.04, then I've tried with symmetric encryption and I've finally seen that it also happens with no encryption at all, when the backup files get stored as .gz files, not .gpg files.

Anyway:

$ gpg --version
gpg (GnuPG) 2.2.4
libgcrypt 1.8.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /home/lemon/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

NOTICE duplicity tests on small size data about 30-40 MBytes were OK apparently, some bigger ones too. Now I was trying to backup my home folder, the backup size is currently about 8.4 GB.

A guess just popping to my mind: maybe it's due to some variable being overflown, as in C when a long integer should be long double integer instead.

Here are some of the lines I used to create the backup in my latest tests. Earlier, I had also tried the --encrypt-sign-key option letting duplicity+gpg figure out which subkeys to use for each function, and I had tried with different keys, each time creating an extra subkey as explained in this tutorial https://www.void.gr/kargig/blog/2013/12/02/creating-a-new-gpg-key-with-subkeys/

I've used 4096 bytes keys.

But I repeat: the problem is present also when no encryption at all is being used.

# ASYMMETRIC
duplicity --full-if-older-than 1M   --encrypt-key 0x2F10ADDB52F43B4E --sign-key 0xE3CD9532B0C19555 --exclude-if-present .notInGeneralBackup --exclude /home/$USER/.bash_history --exclude /home/$USER/.cache     /home/$USER/ file:///media/$USER/MyHardDiskName/BACKUP/home_${USER}__GenBkup_asym

# SYMMETRIC
duplicity --full-if-older-than 1M                                                                  --exclude-if-present .notInGeneralBackup --exclude /home/$USER/.bash_history --exclude /home/$USER/.cache     /home/$USER/ file:///media/$USER/MyHardDiskName/BACKUP/home_${USER}__GenBkup_sym

# NO ENCRYPTION
duplicity --full-if-older-than 1M   --no-encryption                                                --exclude-if-present .notInGeneralBackup --exclude /home/$USER/.bash_history --exclude /home/$USER/.cache     /home/$USER/ file:///media/$USER/MyHardDiskName/BACKUP/home_${USER}__GenBkup_noEnc

Some of the lines I used to verify (earlier I wasn't using the --compare-data option and the SHA1 hash mismatch problem was already appearing):

duplicity verify --compare-data --encrypt-key 0x2F10ADDB52F43B4E --sign-key 0xE3CD9532B0C19555 --exclude-if-present .notInGeneralBackup --exclude /home/$USER/.bash_history --exclude /home/$USER/.cache    file:///media/lemon/Toshiba3TB_SATA3/BACKUP/home_lemon__GenBkup_asym  /home/lemon

duplicity verify --compare-data                                                                --exclude-if-present .notInGeneralBackup --exclude /home/$USER/.bash_history --exclude /home/$USER/.cache    file:///media/lemon/Toshiba3TB_SATA3/BACKUP/home_lemon__GenBkup_sym   /home/lemon

duplicity verify --compare-data                                                                --exclude-if-present .notInGeneralBackup --exclude /home/$USER/.bash_history --exclude /home/$USER/.cache    file:///media/lemon/Toshiba3TB_SATA3/BACKUP/home_lemon__GenBkup_noEnc /home/lemon

duplicity verify --compare-data --no-encryption                                                --exclude-if-present .notInGeneralBackup --exclude /home/$USER/.bash_history --exclude /home/$USER/.cache    file:///media/lemon/Toshiba3TB_SATA3/BACKUP/home_lemon__GenBkup_noEnc /home/lemon

Searching this mailing list archive, I find messages as old as 2003 reporting about some SHA1 hash mismatch problem.
However I also read here and there on the web from various enthusiasts users of duplicity.

Any hint will be much appreciated.