Re: [Duplicity-talk] What is the process for creating signature files ? [DOC-1]

[Kenneth Loafman]

Cyril Russo wrote:
> Cyril Russo a écrit :
>>
>>
>> Hi,
>>
>>  If you have a bit of time, can you explain in few lines how (and
>> where in the code) the signature files are created ?
>>
>> I'm trying to split the signatures to a specified volume size, but I
>> don't want to break anything, and a grep on the code with "signature"
>> is very verbose.
>> Sincerely,
>> Cyril
>>
> Ok, I've tried to sort this out by myself, but I'm still a bit lost.
> Here's what I've understood so far (I hope it could be copy and pasted
> to a documentation file once corrected):
> 
> *Organization of a backup with Duplicity*
>     A collection contains a set of backup volumes (full and incremental
> ones)
>     Each backup volume contains (usually) multiple files (the backup's
> tar archive, a signature file and a manifest file).
> 
> To monitor/inspect a collection, the code is in collections.py which
> declares:
> /BackupSet /(private to collections.py, only used in BackupChain)
> /     /It's the index object for a backup volume.
> /     /It contains the list of files (to) backup(ed), the backup time
> and the stored time range.
>      A backup set is considered complete if the manifest file is found.
>      
> /BackupChain /(private to collections.py, only used in CollectionsStatus)
> /    /It's the index for the list of backup sets.
>     It contains the list of backup set used to recreated the collection,
> starting from a unique (full) backup, and with the incremental backup
> set from then.
> 
> /SignatureChain /(private to collections.py, only used in CollectionsStatus)
>     It's the index of signatures file
>     It contains the first signature (the full signature) and the
> incremental ones.
> 
> /CollectionsStatus /(private to collections.py, used in duplicity-bin)
>     Introspective class used to list/manipulate all BackupChain and
> SignatureChain
> 
> *Backup storage*
>     Backup's volume are stored on a backend's accessible storage as
> named items.
>     Their name is made of the following pattern "duplicity-full" or
> "duplicity-inc" then the UTC date in YYYY-MM-DDTHH:MM:SS, then for
> actual data "volXXX.difftar", or "manifest" for the manifest, or
> "signature" for the signature and finally "gpg".
>   All items are "dot" separated.
> 
>     The backup storage is accessed by the backends, that abstract the
> actual mean of access (SSH, IMAP, FTP, Local FS, etc...).
>     Currently, the backends provide a way to list the items they
> contain, and such items' name are parsed in order to build the
> CollectionsStatus object.
> 
> While backing up, duplicity is using the code from diffdir.py (explained
> in DOC-2 next email).
> Please fix any error in this email so I can continue the backup process
> documentation.
> 
> My goal is to find an elegant way to split the signature file, as
> backends have file size constraint
> Currently, from what I've understood, we could insert a volume number in
> the signature too (creating a SignatureSet quite similar to BackupSet
> class).
> This would add a level of indirection in the signature code.
> Let me know what you think about this.

Looking good so far.

Note: The time format has changed to YYYYMMDDTHHMMSSZ in 0.5.10.

The big issue is not in adding a volume number to the signature file,
but in staying backwards compatible to those that do not have a volume
number.  That should be easy enough.

As to the file format, both the sig and the diff files use tar file
format (tarfile.py), so splitting them should be done in such a way that
a sigtar file is created with each difftar and is completely in parallel
with the data.  What I want to do soon is add another file type called
'partar' file that runs in parallel with the diff and sig tar files and
can act to recover either, using par2, in the case of data corruption.

Thanks for digging into this!  Once its done, we can put it on the web
as part of the documentation that's so sorely needed.  I'll try to be a
bit more timely in my responses.

...Ken

signature.asc
Description: OpenPGP digital signature