Re: [Duplicity-talk] SHA1 hash mismatch (I see previous messages as old as 2003)

[edgar . soldin]

On 07.04.2019 04:46, M. Ini via Duplicity-talk wrote:
> I would /really/ like to be able to use duplicity, but apparently there is a 
> problem. *Any time, some _SHA1 hash mismatch_ error can appear*, even when 
> verifying a backup right after having created it.
>
> I'm storing the backup files into an external USB hard disk which is not 
> giving any problems even with tar archives of various GB each.
>
> I'm on*Ubuntu 18.04*, updated almost daily, I installed the latest stable 
> package for *duplicity****version 0.7.18.2 **from this PPA**: 
> https://launchpad.net/~duplicity-team/+archive/ubuntu/ppa
> *(I had tried at first the version currently available through the default 
> repos configured in Ubuntu 18.04, duplicity 0.7.17.x)
>
> I was at first suspecting a gpg/libcrypt problem due to the fact that I was 
> using keys created on another OS with different GPG/libcrypt versions, so I 
> tried with keys created on Ubuntu 18.04, then I've tried with symmetric 
> encryption and I've finally seen that *it also happens with no encryption at 
> all*, when the backup files get stored as .gz files, not .gpg files.
>
> Anyway:
>
> $ gpg --version
> gpg (GnuPG) 2.2.4
> libgcrypt 1.8.1
> Copyright (C) 2017 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
>
> Home: /home/lemon/.gnupg
> Supported algorithms:
> Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
> Cipher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
>         CAMELLIA128, CAMELLIA192, CAMELLIA256
> Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
> Compression: Uncompressed, ZIP, ZLIB, BZIP2
>
> *
> *
>
> *NOTICE* duplicity tests on small size data about 30-40 MBytes were OK 
> apparently, some bigger ones too. Now I was trying to backup my home folder, 
> the backup size is currently about 8.4 GB.
>
> A guess just popping to my mind: *maybe it's due to some variable being 
> overflown, as in C when a _long_ integer should be _long double_ integer 
> instead*.
>
>
> Here are some of the lines I used to create the backup in my latest tests. 
> Earlier, I had also tried the --encrypt-sign-key option letting duplicity+gpg 
> figure out which subkeys to use for each function, and I had tried with 
> different keys, each time creating an extra subkey as explained in this 
> tutorial 
> https://www.void.gr/kargig/blog/2013/12/02/creating-a-new-gpg-key-with-subkeys/
>
> I've used 4096 bytes keys.
>
> But *I repeat**: the problem is present also when no encryption at all is 
> being used*.
>
>
> # ASYMMETRIC
> duplicity --full-if-older-than 1M   --encrypt-key 0x2F10ADDB52F43B4E 
> --sign-key 0xE3CD9532B0C19555 --exclude-if-present .notInGeneralBackup 
> --exclude /home/$USER/.bash_history --exclude /home/$USER/.cache     
> /home/$USER/ 
> file:///media/$USER/MyHardDiskName/BACKUP/home_${USER}__GenBkup_asym
>
> # SYMMETRIC
> duplicity --full-if-older-than 1M                                             
>                      --exclude-if-present .notInGeneralBackup --exclude 
> /home/$USER/.bash_history --exclude /home/$USER/.cache     /home/$USER/ 
> file:///media/$USER/MyHardDiskName/BACKUP/home_${USER}__GenBkup_sym
>
> # NO ENCRYPTION
> duplicity --full-if-older-than 1M   --no-encryption                           
>                      --exclude-if-present .notInGeneralBackup --exclude 
> /home/$USER/.bash_history --exclude /home/$USER/.cache     /home/$USER/ 
> file:///media/$USER/MyHardDiskName/BACKUP/home_${USER}__GenBkup_noEnc
>
>
> Some of the lines I used to verify (earlier I wasn't using the --compare-data 
> option and the SHA1 hash mismatch problem was already appearing):
>
> duplicity verify --compare-data --encrypt-key 0x2F10ADDB52F43B4E --sign-key 
> 0xE3CD9532B0C19555 --exclude-if-present .notInGeneralBackup --exclude 
> /home/$USER/.bash_history --exclude /home/$USER/.cache    
> file:///media/lemon/Toshiba3TB_SATA3/BACKUP/home_lemon__GenBkup_asym  
> /home/lemon
>
> duplicity verify --compare-data                                               
>                  --exclude-if-present .notInGeneralBackup --exclude 
> /home/$USER/.bash_history --exclude /home/$USER/.cache    
> file:///media/lemon/Toshiba3TB_SATA3/BACKUP/home_lemon__GenBkup_sym   
> /home/lemon
>
> duplicity verify --compare-data                                               
>                  --exclude-if-present .notInGeneralBackup --exclude 
> /home/$USER/.bash_history --exclude /home/$USER/.cache    
> file:///media/lemon/Toshiba3TB_SATA3/BACKUP/home_lemon__GenBkup_noEnc 
> /home/lemon
>
> duplicity verify --compare-data --no-encryption                               
>                  --exclude-if-present .notInGeneralBackup --exclude 
> /home/$USER/.bash_history --exclude /home/$USER/.cache    
> file:///media/lemon/Toshiba3TB_SATA3/BACKUP/home_lemon__GenBkup_noEnc 
> /home/lemon
>
>
> *Searching this mailing list archive, I find messages as old as 2003 
> reporting about some SHA1 hash mismatch problem.*
> However I also read here and there on the web from various enthusiasts users 
> of duplicity.
>
> *Any hint will be much appreciated.*
>

hey M,

1.
please provide a full error stack. run duplicity with max. verbosity '-v9' and 
provide the full output of the terminal.

2.
what's your python version?

generally a checksum error is a protective measure that checks if backend files 
are still the same during verify/restore. if you regularily run into those i 
could imagine that maybe your python version has a bug.

currently there is no known checksum error issue. check the mailing list again 
and you will probably see that issues were rooted in bad file systems or 
transfer problems etc.

..ede/duply.net