[rdiff-backup-users] Hidden directories are ignored during incremental backup

[Cybertinus]

Hello everybody,

This is my first post to this mailing list, so I hope I don't ask to many stupid 
questions ;). And also a short introduction:
I'm Cybertinus, a 22 year old IT-student from The Netherlands. I tend to fiddle 
around a bit with computers at home. Just playing a bit with Linux server aimed 
for server usage. Applications like Apache and MySQL, but also Nagios or Snort. 
Just to find out what it takes to get a secure, stable server up and running. 
And one of the most important parts of secure and stable is backups. And at my 
desktop at home I use rdiff-backup for the backups.
And when I create a new topic on a mailinglist or forum, I always seam to manage 
that is becomes a very long post :p, sry for that ;).

First let me explain what my backup plan is :) (you can skip this paragraph if 
you are only interested in the problem I have ;) ). I have 3 external 
harddrives. The first one is mounted on /backup and rdiff-backup writes the 
complete backup to that directory (well, /backup/snapshots actually).
The second disk is also connected to my computer, but then at the mountpoint 
/backup_sync. 15 minutes after rdiff-backup has updated the backup on /backup I 
use rsync to copy everything from /backup to /backup_sync (why? I tell you in a 
bit ;) )
The third disk is at my grandma's place, for external backup. Even when my house 
burns down, my data is safe :).
Once every week I disconnect the disk which was mounted at /backup and take that 
to my grandma. The disk that was there I take back home with me, and gets 
connected to /backup_sync. The disk that was at /backup_sync already moved to 
/backup, when the original disk at /backup was disconnected. This way I can 
rotate my disks and can I make incremental backups at my computer, and I have 
all the time in the world to swap the disks at my grandma's place :).
I've written a little script to automate the backup. I've created 3 backupsets: 
the first set backups my most important directories every hour to my external 
disk, the first set backups the little bit less important directories every day 
and the rest of the data I want to keep gets backuped in the third set, once a week.

My problem I have with rdiff-backup is that it seams to ignore hidden files 
(files or directories that start with a .) when it creates a incremental backup. 
I use Courier IMAP as a mailserver and the mails are stored in a maildir. So 
I've got directories like ~/mail/.CybertinusNl.site1, 
~/mail/.CybertinusNl.site2, etc. But those directories don't get incremented 
when I run rdiff-backup again. I've added a new maildir today 
(~/mail/.CybertinusNl.Backup) but that directory doesn't show up in the output 
directory of rdiff-backup.

I've written a script to make the configuration an generation of my backups very 
easy. This scripts is actually multiple files. The main file I called 
rdiff-main, and I've placed in /usr/sbin. It looks like this (don't mind the 
comments in the beginning, it is Dutch text, explaining the meaning of the 
script and stuff, nothing important):
#!/bin/bash

# Handige optie om incrementeel te backuppen...
# local of remote (default via ssh).
# Ik vind het handiger dan rsync of rsnapshot.

# Home: http://www.nongnu.org/rdiff-backup
# Voor Debian: in de repository
# Voor Red Hat Enterprise Linux/CentOS: bij Dag Wieers
# Voor Gentoo: in Porage
# Orgineel script: http://www.mrleejohn.nl/downloads/rdiff.txt

DEST="/backup/snapshots"

if [ -x /etc/rdiff-backup/rdiff-$1 ] ; then
    source /etc/rdiff-backup/rdiff-$1
else
    echo "The file rdiff-$1 doesn't exist. Please fix your config" 1>&2
    exit 1
fi

cd /
for dir in $DIRS ; do
    echo "Starting backup of $dir"
    rdiff-backup --create-full-path $dir "$DEST/$1/$dir"
    rdiff-backup --remove-older-than $TIME_SAVED "$DEST/$1/$dir"
    echo "Done backing up $dir"
done

As you can see, the script requires one argument, which defines which file needs 
to be included. That argument can be 'hourly' for example. 
/etc/rdiff-backup/rdiff-hourly looks like this:
#!/bin/bash

export DIRS='/home/mysql /home/svn /home/cybertinus/bash 
/home/cybertinus/localhost /home/cybertinus/mail'
export TIME_SAVED='2D'

So, just a configuration file, nothing fancy. But a neat way to create different 
backupsets ;).
When I first created the backup everything gets copied, but now I need it to 
increment the existing backup, but I don't see anything happening at 
/backup/snapshots/hourly/home/cybertinus/mail. The new directories don't get 
visible.

I've searched the manpage, the wiki and the mailinglist, but nothing told me 
anything about those hidden directories. I found one post on the mailinglist 
that told a user that you can use --exclude '**/.*' to skip all the hidden 
files. So I added --include '**/.*' to my main script, but then I got an error 
that rdiff-backup includes everything right away, so adding --include wasn't 
needed and that I probably meant something else, so the script just exited :(.
I also tried adding --backup-mode (with and without --force), but to no avail, 
the hidden directories still get ignored.
When I look at the directory with rdiff-backup -l 
/backup/snapshots/hourly/home/cybertinus/mail I see that it created a new 
snapshot. And normal files also got updated.

So, am I doing something wrong with rdiff-backup? Or don't I understand what I 
see in /backup/snapshots/hourly/home/cybertinus/mail? The last thing is very 
well possible, I just started using rdiff-backup (I used rsnapshot before this, 
but that wasn't exactly what I needed, and before that I didn't make any backups).

Thnx in advance for every reply :).

Best regards,
Cybertinus