Re: cross-platform backup tool Same files from different source dir causes spurious diff files

[Leland Best]

Hi Cliff,

On Mon, 2022-02-07 at 11:45 -0800, Mr. Clif wrote:
> Hey Eric,
> 
> any ideas on this? How do these diff files normally work?
[...]

I'm not an 'rdiff-backup' developer or anything so all you experts out there
correct me if I'm wrong but ...

IIRC 'rdiff-backup' keeps inode info as part of the metadata for each file. 
When you mount a filesystem Linux assigns "fake" inode numbers to avoid
collisions between filesystems on different devices/partitions/etc..  So if you
change the mount point, every file could potentially get a new inode number and,
consequently, have changed metadata.  That results in 'rdiff-backup' creating a
'*.diff*' file for every source file.

In addition, since 'rdiff-backup' now thinks the files may have changed it
spends a lot of time checking if anything other than metadata has changed which
_might_ account for the apparently low throughput.

Maybe?  Anybody?

Cheers
Leland
-- 
-------------------------------------------------------------------------------
Leland C. Best      | Creationists make it sound as though a 'theory' is
lcbpublic@gmail.com |  something you dreamt up after being drunk all night.
                    | -- Isaac Asimov
PGP Fingerprint     | 7B23ACB474299DF0E07F6DEBB8FEE465DCB9B6C3
-------------------------------------------------------------------------------

[...]
> 
>      Thanks,
>      Clif
> 
> On 2/5/22 11:05 AM, Mr. Clif wrote:
> > Greetings,
> > 
> > Recently I started experimenting with backing up LVM snapshots of live 
> > systems instead of the systems themselves. This protects against 
> > getting complaints about files changing like:
> > 
> > UpdateError: 
> > 'clif/.mozilla/firefox/5zz41jpw.default/sessionstore-backups/recovery.baklz4'
> > Updated mirror temp file 
> > '/leda.bak/home/clif/.mozilla/firefox/5zz41jpw.default/sessionstore-
> > backups/rdiff-backup.tmp.75'
> > does not match source
> > 
> > So for example instead of the source being the root dir of a remote 
> > server, it's now /mnt/vm-backup-snapshot/ on the local server. The 
> > destination is the same. The first time I tried this I got .diff and 
> > .diff.gz files for pretty much every file in the filesystem. The vast 
> > majority of these files are very small like 80-100 bytes. Here is a 
> > sample:
> > 
> > 
> > -rwxr-xr-x 1 root root 96 Feb 28  2019 
> > tank-rdiff/rdiff-backup-data/increments/bin/dir.2022-02-01T05:20:43-
> > 09:00.diff.gz
> > -rwxr-xr-x 1 root root 103 Mar 18  2021 
> > tank-rdiff/rdiff-backup-data/increments/bin/systemctl.2022-02-01T05:20:43-
> > 09:00.diff.gz
> > -rwxr-xr-x 1 root root 96 Feb 28  2019 
> > tank-rdiff/rdiff-backup-data/increments/bin/touch.2022-02-01T05:20:43-
> > 09:00.diff.gz
> > -rwxr-xr-x 1 root root 80 Feb 28  2019 
> > tank-rdiff/rdiff-backup-data/increments/bin/rm.2022-02-01T05:20:43-
> > 09:00.diff.gz
> > -rwxr-xr-x 1 root root 93 Jan 21  2019 
> > tank-rdiff/rdiff-backup-data/increments/bin/run-parts.2022-02-01T05:20:43-
> > 09:00.diff.gz
> > -rwxr-xr-x 1 root root 91 Feb 28  2019 
> > tank-rdiff/rdiff-backup-data/increments/bin/chown.2022-02-01T05:20:43-
> > 09:00.diff.gz
> > -rwxr-xr-x 1 root root 94 Mar 18  2021 
> > tank-rdiff/rdiff-backup-data/increments/bin/journalctl.2022-02-01T05:20:43-
> > 09:00.diff.gz
> > -rwxr-xr-x 1 root root 83 Feb 28  2019 
> > tank-rdiff/rdiff-backup-data/increments/bin/sync.2022-02-01T05:20:43-
> > 09:00.diff.gz
> > -rwxr-xr-x 1 root root 96 Mar 18  2021 
> > tank-rdiff/rdiff-backup-data/increments/bin/systemd-escape.2022-02-
> > 01T05:20:43-09:00.diff.gz
> > -rwxr-xr-x 1 root root 109 Mar 18  2021 
> > tank-rdiff/rdiff-backup-data/increments/bin/systemd-tty-ask-password-
> > agent.2022-02-01T05:20:43-09:00.diff.gz
> > -rwxr-xr-x 1 root root 90 Feb 28  2019 
> > tank-rdiff/rdiff-backup-data/increments/bin/false.2022-02-01T05:20:43-
> > 09:00.diff.gz
> > -rwxr-xr-x 1 root root 67 Jan  5  2019 
> > tank-rdiff/rdiff-backup-data/increments/bin/zforce.2022-02-01T05:20:43-
> > 09:00.diff.gz
> > -rwxr-xr-x 1 root root 65 Jan  5  2019 
> > tank-rdiff/rdiff-backup-data/increments/bin/znew.2022-02-01T05:20:43-
> > 09:00.diff.gz
> > -rwxr-xr-x 1 root root 98 Jan 21  2019 
> > tank-rdiff/rdiff-backup-data/increments/bin/tempfile.2022-02-01T05:20:43-
> > 09:00.diff.gz
> > 
> > The total space used by this increment was around 500MB and the 
> > largest file was (the only file size that had a suffix):
> > 
> > -rw-r----- 1 root adm 2.1K Jan 22 05:21 
> > tank-rdiff/rdiff-backup-data/increments/var/log/auth.log.2022-02-01T05:20:43-
> > 09:00.diff.gz
> > 
> > I looked at some of these diff files and they seemed empty except for 
> > a binary header that I didn't find documentation for in the 
> > rdiff-backup file format section. I also noticed that occasionally, 
> > these "empty" diffs show up in normal day to day increments. I realize 
> > that diffs also can show up when the permissions change but I don't 
> > think all these files changed.
> > 
> > Is this how it's supposed to work, and could you please explain it to 
> > me? :-)
> > 
> >     Thanks,
> >     Clif
> > 
> > 
> 
>