Re: [rdiff-backup-users] atomic increment files?

[Marcel (Felix) Giannelia]

OK, after writing a few little utilities, I was able to investigate this 
further last night and found that the inability to run du is not 
entirely rdiff-backup's fault -- one of the machines that's backing up 
to this server has a misconfigured sendmail that's (long story short) 
causing about 15000 modified files every day in /var/spool/clientmqueue. 
That translates into one million files in the same directory in 
rdiff-backup's increments (accumulated over 154 runs), so it's no wonder 
everything that does directory listings is choking there.

However, I stand by what I said earlier -- making thousands of little 
files on purpose is a bad design.

In case anyone else needs a way to quickly estimate the size of an 
rdiff-backup target without traversing the filesystem, see 
rdiff-backup-size (in the contribscripts section of the wiki) -- it's a 
quick & dirty Perl script I wrote that just adds up the sizes in the 
session_statistics files.

~Felix.



On 07/03/09 15:03, Marcel (Felix) Giannelia wrote:
> Is there any way of making rdiff-backup produce single files as 
> increments (say, by zipping them together when it produces them), 
> instead of thousands of itty bitty files? One file per increment would 
> make the task of moving old ones onto archive DVD a lot easier, and 
> would make a lot less hardship for the target machine's filesystem, 
> too. It probably wouldn't slow down restores all that much, as 
> accessing an archive file's directory structure is likely faster than 
> doing the same in a part of the filesystem containing many thousands 
> of files per directory.
>
> Presently, I'm trying to do a du -s on our backup directories and it 
> sat there for over an hour without having printed the size of the 
> first one. According to top, du was using 50% of the total memory. I 
> know that there are statistics files which I could add together, but 
> in this case I want to use du to be sure because there's a chance that 
> there might be stray files in our rdiff-backup-data directory. Also, 
> creating so many files that commands like du cannot even function is, 
> in my opinion, incorrect behaviour.
>
> ~Felix.
>
>
> _______________________________________________
> rdiff-backup-users mailing list at address@hidden
> http://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> Wiki URL: 
> http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBackupWiki