Re: [rdiff-backup-users] Performance

[Greg Freemyer]

On 2/13/07, Dave Howorth <address@hidden> wrote:
> Jim C. Nasby wrote:
> > On Sat, Feb 10, 2007 at 10:15:53PM -0700, Corey Puffalt wrote:
> >> Is anyone else having any performance issues with rdiff-backup?
> >
> > I've run into numerous cases where rdiff-backup just doesn't perform
> > well, especially over a network. For example, doing a restore I'm doing
> > good to see 800kB/s (according to Activity Monitor in OS X), while an
> > scp of the same file will do 2+ MB/s.
> >
> > Reverting a failed backup is also very slow. As is listing the size of
> > increments.
>
> Does anybody know of an alternative that performs better?
>
> I have a filesystem with millions of hard links and so far rdiff-backup
> is the fastest thing I've found, even though it is slow.
>
> Cheers, Dave

Dave,

If you have millions of anything, then the filesystem selection
probably matters more than the backup software.  Reiser3 is known to
have been specifically tuned to handle huge numbers of small files.
(It was tuned to be a imap mail server with each msg being a
standalone file.).

Also, the layout of your files/directories matters a lot.  I have
experience with that under NTFS, but in a NTFS setup you really want
to stay below 100K files per directory and children of the directory.
ie. When the windows file explorer enters a directory it goes ahead
and opens any directories within the main directory.  It then attempts
to cache some info about all of that.  If the number of cummulative
files exceeds 100K you see a drastic slowdown.

Greg
--
Greg Freemyer
The Norcross Group
Forensics for the 21st Century