Re: [Duplicity-talk] What is the process for creating signature files ? [DOC-1]

[Cyril Russo]

Cyril Russo a écrit :

Hi,

 If you have a bit of time, can you explain in few lines how (and where in the code) the signature files are created ?

I'm trying to split the signatures to a specified volume size, but I don't want to break anything, and a grep on the code with "signature" is very verbose.
Sincerely,
Cyril

Ok, I've tried to sort this out by myself, but I'm still a bit lost.
Here's what I've understood so far (I hope it could be copy and pasted to a documentation file once corrected):

Organization of a backup with Duplicity
    A collection contains a set of backup volumes (full and incremental ones)
    Each backup volume contains (usually) multiple files (the backup's tar archive, a signature file and a manifest file).

To monitor/inspect a collection, the code is in collections.py which declares:
BackupSet (private to collections.py, only used in BackupChain)
     It's the index object for a backup volume.
     It contains the list of files (to) backup(ed), the backup time and the stored time range.
     A backup set is considered complete if the manifest file is found.
     
BackupChain (private to collections.py, only used in CollectionsStatus)
    It's the index for the list of backup sets.
    It contains the list of backup set used to recreated the collection, starting from a unique (full) backup, and with the incremental backup set from then.

SignatureChain (private to collections.py, only used in CollectionsStatus)
    It's the index of signatures file
    It contains the first signature (the full signature) and the incremental ones.

CollectionsStatus (private to collections.py, used in duplicity-bin)
    Introspective class used to list/manipulate all BackupChain and SignatureChain

Backup storage
    Backup's volume are stored on a backend's accessible storage as named items.
    Their name is made of the following pattern "duplicity-full" or "duplicity-inc" then the UTC date in YYYY-MM-DDTHH:MM:SS, then for actual data "volXXX.difftar", or "manifest" for the manifest, or "signature" for the signature and finally "gpg".
  All items are "dot" separated.

    The backup storage is accessed by the backends, that abstract the actual mean of access (SSH, IMAP, FTP, Local FS, etc...).
    Currently, the backends provide a way to list the items they contain, and such items' name are parsed in order to build the CollectionsStatus object.

While backing up, duplicity is using the code from diffdir.py (explained in DOC-2 next email).
Please fix any error in this email so I can continue the backup process documentation.

My goal is to find an elegant way to split the signature file, as backends have file size constraint
Currently, from what I've understood, we could insert a volume number in the signature too (creating a SignatureSet quite similar to BackupSet class).
This would add a level of indirection in the signature code.
Let me know what you think about this.