[rdiff-backup-users] Memory problems - any suggestions?

[David]

Hi, rdiff-backup is using excessive amounts of memory on my system
(almost 2 GB used on my system with 1 GB RAM and 1 GB swap).

I've checked the FAQ - My rdiff-backup version is 1.1.5-4, and
librsync1 is 0.9.7-1. These are Debian versions, I'm running
rdiff-backup on a Debian Etch system.

More information about my usage:

I'm using a combination of rsync and rdiff-backup to preserve history
for millions of files (a backup server which backs up another backup
server).

For the sake of explanation:

backup1 - original backup server (pulls data from workstations, other
servers, etc)
backup2 - backup server which pulls data from backup1 and keeps history.

backup1 also uses a combination of rsync and rdiff-backup. So backup2
will be backing up rdiff-backup metadata from backup1:

I use rdiff-backup to get the history, but use rsync to fetch the
files (because it's easy to install DeltaCopy on Windows boxes. Also,
rsync has extra options I like to use).

I have a slightly convoluted backup logic - this is necessary as I
need to preserve disk space on the backup server (can't have a
temporary copy that is the same size as the backup store).

My logic goes something like this (on backup2):

1) I start off with a directory structure like this:

/data/backup/files/ - Backups are stored here
/data/backup/files/rdiff-backup-data - rdiff-backup stores it's
metadata and history here

2) Make a temporary work directory:

/data/backup/new/

I create this directory with rsync - i tell it to create hardlinks to
the files under /data/backup/files/, but excluding the
rdiff-backup-data sub-directory

3) Use rsync to pull files from backup1 and update /data/backup/new/
(also deleting files which are no longer under /data/backup/new/)

Also, I preserve hardlinks (and numeric user IDs) at this stage,
because some of the backups are of servers and I want to be able to
restore them with rsync, preserving hardlinks, users, groups, etc.

4) Use rdiff backup to push any changes from /data/backup/new/ to
/data/backup/files, and create history.

rdiff-backup gets a bit confused here because of occasionall-changing
inodes, so I give it a few options:

 --preserve-numerical-ids --no-compare-inode --force

This is also the step where rdiff-backup uses excessive memory.

5) Remove the temporary /data/backup/new/ directory.

So, there you have it. Does anyone have suggestions for:

1) Improving my somewhat complicated backup system
2) Using rdiff-backup better, so I don't have to override it's inode warnings
3) Solving the memory problem?

Thanks in advance.

David.