[Duplicity-talk] Considering a replacement for the tar file format for backups

[Naja Melan]

Hi,

Since I'm working on a backup utility myself I have been enthusiast to discover 
duplicity and your proposition for an alternative backup format. I would like 
to share some first impressions about the proposed format:

- rather than using gpg for encryption a more block-device type of encryption 
would be desirable. Otherwise you fall in the same trap as with tar (I refer to 
the disadvantages of tar you sum up). The meta data might have confidential 
information, so it should be encrypted. It would be possible to encrypt it 
separately, but it would be much simpler if the whole file would be encrypted 
but support random access.

- I like the overall structure and probably some standards for the archive 
header could be made which would allow different backup utilities to understand 
the same files. But then again why reinvent the wheel. What about something 
like a truecrypt file container (maybe formatted to the same file system as the 
source files reside on) holding some backup specific meta files. Note that this 
immediately solves the problem of extra file attributes and permissions and the 
like, but also restricts putting files from different sources in the same 
backup. Maybe it could create quite some overhead for small files. Actual files 
on the volume could be compressed with whatever algorithm, benefiting from the 
error correction present in existing archive formats or file systems.

- considering config files I'm starting to get convinced of yaml. Advantages 
include: next to no overhead, easily readable and somehow writeable by humans. 
Disadvantages are mainly it's badly documented and does not allow freedom over 
the whitespace which makes it kind of tricky to understand all the intricacies 
for a human writer.

In brief, move your encryption to the outer layer so you guarantee privacy, 
next have a file system, and do the compression on the last level to avoid 
wasting cycles on compressing already compressed formats like jpg, mp3 etc...

greets,
naja melan