Re: [rdiff-backup-users] rdiff-backup result is missing text files but no errors, sourcedir to big >2TB

[Dominic Raferd]

On Fri, 19 Jul 2019 at 20:43, Jelle de Jong <address@hidden>
wrote:

> On 7/18/19 4:45 PM, Lew Wolfgang wrote:
> > On 07/17/2019 04:48 AM, Jelle de Jong wrote:
> >> Hello everybody,
> >>
> >> I am trying to run an rdiff-backup and it keeps missing some documents
> >> compared with the source, we are using a simple test.txt file to check
> >> if the backups are still working and I got one big source backup of
> >> +2.4TB. When running rdiff-backup there are no errors. So I can not
> >> figure out what is going wrong.
> >>
> >> I split the backup to make the source smaller and then it does make
> >> backups of the test.txt file.
> >>
> >> How can rdiff-backup handle bigger 3TB backups? The command I am
> running?
> >>
> >> /usr/bin/python /usr/bin/rdiff-backup --tempdir
> >> /srv/storage/backups/tempdir --exclude-device-files --exclude-sockets
> >> --exclude-fifos /mnt/sr8-sdb2/
> >> /srv/storage/backups/libvirt-filesystems/sr8-sdb2/
> >
> > Just a wild guess:  a size limitation in your disk partition,
> > filesystem, or
> > operating system?
> >
> > What operating system:
> > 32 or 64-bits:
> > Partition table:  (MBR or GPT)
> > Filesystem type:  (NTFS, EXT4, ZFS, XFS, etc)
> > Size of destination partition:
> > BIOS type:  (Legacy or UEFI)
>
> Thank you all for responding, here is the information.
>
> # rdiff-backup --version
> rdiff-backup 1.2.8
>
> root@backup:~# uname -a
> Linux backup 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5 (2019-06-19) x86_64
> GNU/Linux
>
> root@backup:~# df -hal /srv/storage/backups/libvirt-filesystems/sr8-sdb2
> Filesystem                     Size  Used Avail Use% Mounted on
> /dev/mapper/lvm0--vol-storage   22T   15T  7.6T  66% /srv/storage
>
> root@backup:~# mount | grep  /dev/mapper/lvm0--vol-storage
> /dev/mapper/lvm0--vol-storage on /srv/storage type xfs
> (rw,relatime,attr2,inode64,logbsize=64k,sunit=128,swidth=1024,noquota)
>
> root@backup:~# mount | grep /boot
> /dev/sdr1 on /boot type ext3 (rw,relatime)
>
> root@backup:~# parted /dev/sdr1 print
> Model: Unknown (unknown)
> Disk /dev/sdr1: 30.7GB
> Sector size (logical/physical): 512B/512B
> Partition Table: loop
> Disk Flags:
>
> Number  Start  End     Size    File system  Flags
>   1      0.00B  30.7GB  30.7GB  ext3
>
> # Legacy bios
> root@backup:~# dmidecode -t bios
> # dmidecode 3.2
> Getting SMBIOS data from sysfs.
> SMBIOS 2.5 present.
>
> Handle 0x0005, DMI type 0, 24 bytes
> BIOS Information
>         Vendor: Intel Corp.
>         Version: S5500.86B.01.00.0061.030920121535
>         Release Date: 03/09/2012
>         Address: 0xF0000
>         Runtime Size: 64 kB
>         ROM Size: 8192 kB
>         Characteristics:
>                 PCI is supported
>                 PNP is supported
>                 BIOS is upgradeable
>                 BIOS shadowing is allowed
>                 Boot from CD is supported
>                 Selectable boot is supported
>                 EDD is supported
>                 3.5"/2.88 MB floppy services are supported (int 13h)
>                 Print screen service is supported (int 5h)
>                 8042 keyboard services are supported (int 9h)
>                 Serial services are supported (int 14h)
>                 CGA/mono video services are supported (int 10h)
>                 ACPI is supported
>                 USB legacy is supported
>                 LS-120 boot is supported
>                 ATAPI Zip drive boot is supported
>                 Function key-initiated network boot is supported
>                 Targeted content distribution is supported
>         BIOS Revision: 17.18
>         Firmware Revision: 0.0
>
> Handle 0x002D, DMI type 13, 22 bytes
> BIOS Language Information
>         Language Description Format: Long
>         Installable Languages: 1
>                 en|US|iso8859-1
>         Currently Installed Language: en|US|iso8859-1
>
> /usr/bin/python /usr/bin/rdiff-backup --tempdir
> /srv/storage/backups/tempdir --exclude-device-files --exclude-sockets
> --exclude-fifos /mnt/sr8-sda2
> /srv/storage/backups/libvirt-filesystems/sr8-sda2
>
> root@backup:~# dd if=/mnt/sr8-sda2/pagefile.sys
> of=/srv/storage/backups/tempdir/pagefile.sys
> 2621440+0 records in
> 2621440+0 records out
> 1342177280 bytes (1.3 GB, 1.2 GiB) copied, 11.8273 s, 113 MB/s
>
> Is there an option to speed up /usr/bin/rdiff-backup it runs with about
> 01-10M/s compared to the possible 113M/s on the underlying systems.


Could there be some permission / file lock issues on the source? Have you
tried creating an LVM snapshot of the source and backing up from that
instead?

Regarding speed, you can try turning off compression in rdiff-backup with
the --no-compression switch.