Re: [Duplicity-talk] Restoring from PCA Backend - download preparation

[Kenneth Loafman]

Hi Erwan,

I think you are on the right track, except, your changes need to be in the backends, no place else.  The way this works is that if the backend implements pre_proccess_download[_batch], then they will be used by duplicity.  The way to go is to implement those two functions in your backends.  duplicity will not be confused since it checks with hasattr() to see if it's available.

We implement site-specific issues in the backend that handles the site.  It keeps the base code clean that way.

...Thanks,

...Ken

On Sun, Jun 27, 2021 at 5:05 AM Erwan via Duplicity-talk <duplicity-talk@nongnu.org> wrote:

Hi,

 

I’m using duplicity with multibackend (public cloud archive and object storage from ovh), as described in this documentation from ovh :

https://docs.ovh.com/gb/en/storage/pca/duplicity/

I’ve worked on a modification to make PCA Backend work better (for my usage at least !) : allow to unseal all volumes at once when restoring content from PCA, wait for unseal to complete for all volumes before launching download. I’ve used API documentation as mentioned here :

https://docs.ovh.com/gb/en/storage/pca/api/

 

This is a similar change to what has been implemented some time ago for Glacier cold storage :

https://lists.nongnu.org/archive/html/duplicity-talk/2020-07/msg00021.html

I’ve implemented the following things available in this commit : https://gitlab.com/erwanBoka/duplicity/-/commit/822e42ba3c26a5b32377e238a9b6040b9b087052

• Implement in BackendWrapper and MultiBackend pre_process_download (and redirect to sub-backends)
• Implement for pcabackend pre_process_download with ovh API as a blocking call, waiting for all volumes to be unsealed.

 

However, I’m a bit confused by what was implemented then (new hook pre_process_download_batch), compared to existing one pre_process_download.

• pre_process_download is called from dup_main/restore_get_patched_rop_iter with all filenames as parameters
• it’s implemented only once in _boto_single, but with a single filename as argument…
• pre_process_download_batch is essentially doing the same thing (call with all volume names)

I’m afraid my modifications in BackendWrapper and MultiBackend might cause pre_process_download to be called for _boto_single, and break things…

 

I haven’t modified anything yet in _boto_single, but my proposal would be :

• change pre_process_download method name in _boto_single to an « internal » method used only in _get
• keep only pre_process_download hook in dup_main
• implement it the way pre_process_download_batch is implemented today in _boto_single

I don’t have s3 access, and cannot test it though.

 

Let me know your view about this, and the path forward please !

Best regards,

Erwan

 

 

 

 

_______________________________________________
Duplicity-talk mailing list
Duplicity-talk@nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk