Re: [rdiff-backup-users] rdiff-backup result is missing text files but no errors, sourcedir to big >2TB

[Jelle de Jong]

On 7/18/19 4:45 PM, Lew Wolfgang wrote:
> On 07/17/2019 04:48 AM, Jelle de Jong wrote:
> > Hello everybody,
> >
> > I am trying to run an rdiff-backup and it keeps missing some documents 
> > compared with the source, we are using a simple test.txt file to check 
> > if the backups are still working and I got one big source backup of 
> > +2.4TB. When running rdiff-backup there are no errors. So I can not 
> > figure out what is going wrong.
> >
> > I split the backup to make the source smaller and then it does make 
> > backups of the test.txt file.
> >
> > How can rdiff-backup handle bigger 3TB backups? The command I am running?
> >
> > /usr/bin/python /usr/bin/rdiff-backup --tempdir 
> > /srv/storage/backups/tempdir --exclude-device-files --exclude-sockets 
> > --exclude-fifos /mnt/sr8-sdb2/ 
> > /srv/storage/backups/libvirt-filesystems/sr8-sdb2/ 
>
> Just a wild guess:  a size limitation in your disk partition, 
> filesystem, or
> operating system?
>
> What operating system:
> 32 or 64-bits:
> Partition table:  (MBR or GPT)
> Filesystem type:  (NTFS, EXT4, ZFS, XFS, etc)
> Size of destination partition:
> BIOS type:  (Legacy or UEFI)

Thank you all for responding, here is the information.

# rdiff-backup --version
rdiff-backup 1.2.8

root@backup:~# uname -a
Linux backup 4.19.0-5-amd64 #1 SMP Debian 4.19.37-5 (2019-06-19) x86_64 
GNU/Linux

root@backup:~# df -hal /srv/storage/backups/libvirt-filesystems/sr8-sdb2
Filesystem                     Size  Used Avail Use% Mounted on
/dev/mapper/lvm0--vol-storage   22T   15T  7.6T  66% /srv/storage

root@backup:~# mount | grep  /dev/mapper/lvm0--vol-storage
/dev/mapper/lvm0--vol-storage on /srv/storage type xfs 
(rw,relatime,attr2,inode64,logbsize=64k,sunit=128,swidth=1024,noquota)

root@backup:~# mount | grep /boot
/dev/sdr1 on /boot type ext3 (rw,relatime)

root@backup:~# parted /dev/sdr1 print
Model: Unknown (unknown)
Disk /dev/sdr1: 30.7GB
Sector size (logical/physical): 512B/512B
Partition Table: loop
Disk Flags:

Number  Start  End     Size    File system  Flags
 1      0.00B  30.7GB  30.7GB  ext3

# Legacy bios
root@backup:~# dmidecode -t bios
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 2.5 present.

Handle 0x0005, DMI type 0, 24 bytes
BIOS Information
        Vendor: Intel Corp.
        Version: S5500.86B.01.00.0061.030920121535
        Release Date: 03/09/2012
        Address: 0xF0000
        Runtime Size: 64 kB
        ROM Size: 8192 kB
        Characteristics:
                PCI is supported
                PNP is supported
                BIOS is upgradeable
                BIOS shadowing is allowed
                Boot from CD is supported
                Selectable boot is supported
                EDD is supported
                3.5"/2.88 MB floppy services are supported (int 13h)
                Print screen service is supported (int 5h)
                8042 keyboard services are supported (int 9h)
                Serial services are supported (int 14h)
                CGA/mono video services are supported (int 10h)
                ACPI is supported
                USB legacy is supported
                LS-120 boot is supported
                ATAPI Zip drive boot is supported
                Function key-initiated network boot is supported
                Targeted content distribution is supported
        BIOS Revision: 17.18
        Firmware Revision: 0.0

Handle 0x002D, DMI type 13, 22 bytes
BIOS Language Information
        Language Description Format: Long
        Installable Languages: 1
                en|US|iso8859-1
        Currently Installed Language: en|US|iso8859-1

/usr/bin/python /usr/bin/rdiff-backup --tempdir 
/srv/storage/backups/tempdir --exclude-device-files --exclude-sockets 
--exclude-fifos /mnt/sr8-sda2 
/srv/storage/backups/libvirt-filesystems/sr8-sda2

root@backup:~# dd if=/mnt/sr8-sda2/pagefile.sys 
of=/srv/storage/backups/tempdir/pagefile.sys
2621440+0 records in
2621440+0 records out
1342177280 bytes (1.3 GB, 1.2 GiB) copied, 11.8273 s, 113 MB/s

Is there an option to speed up /usr/bin/rdiff-backup it runs with about 
01-10M/s compared to the possible 113M/s on the underlying systems.

Kind regards,

Jelle de Jong