Re: [Duplicity-talk] Performance issues w/ S3

[tink]

Hi Thomas, All,

Thank you so much for all the effort and sorry for my prolonged silence.

I have now tested w/ aws cli, Thoma's python snippet and duplicity (versions duplicity 0.8.22 on Debian 12, duplicity 0.8.12 on Ubuntu 20.04 and duplicity 2.2.3 on alpine) ... in all cases aws cli is fastest, the python code somewhat  

slower (adding some 35% transfer time) and duplicity crawling & stalling .... a 10 G transfer that took 2 minutes to wasabi using aws cli taking over 15+/-5 minutes in each instance. 

Less stalling against aws than against wasabi, but still ... 

On Mon, 15 Jul 2024 at 01:41, Thomas Laubrock via Duplicity-talk <duplicity-talk@nongnu.org> wrote:

Hi Ede

Am 13.07.24 um 18:11 schrieb edgar.soldin--- via Duplicity-talk:
> maybe it's the multi_chunking? duplicity's s3 backend uses some
> default config vars
> https://gitlab.com/duplicity/duplicity/-/blob/dev/duplicity/config.py?ref_type=heads#L246-252


As written the uploads saturating my upload limits, therefore I don't
think that this will have heavy impact.

Netto upload times from pur bot3 and duplicity are the same.

>
>
> to set up a custom TransferConfig and finally use upload_file()
> https://gitlab.com/duplicity/duplicity/-/blob/dev/duplicity/backends/s3_boto3_backend.py?ref_type=heads#L164-186
>
>
> afaiu the chunking only happens if the file is bigger than the given
> chunk size. so maybe this should be lowered or the test file expanded 🙂


Me and Tink using the default Volume Size of 200 MB, chunks are 20MB by
default

https://gitlab.com/duplicity/duplicity/-/blob/dev/duplicity/config.py#L243-252


Thoses setting sound pretty reasonable to me. I don't think that
chunking will have heavy impact here.

The boto3 default chunk size is a 8MB

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html


Bye

  Thomas



_______________________________________________
Duplicity-talk mailing list
Duplicity-talk@nongnu.org
https://lists.nongnu.org/mailman/listinfo/duplicity-talk

--

A: Because it messes up the order in which people normally read text.

Q: Why is top-posting such a bad thing?

A: Top-posting.

Q: What is the most annoying thing in e-mail?