Re: [Duplicity-talk] Performance issues w/ S3

[tink]

Hi Thomas,

On Mon, 22 Jul 2024 at 21:57, Thomas Laubrock via Duplicity-talk <duplicity-talk@nongnu.org> wrote:

Hi Tink,

Am 21.07.24 um 23:21 schrieb tink via Duplicity-talk:

And without compression and encryption the local version takes 55 seconds.

The compression and encryption is done by feeding the data into GnuPG binary installed on your system.

I suggest following tests:

- use `gpg` and encrypt a file manually and check the timing.

Encrypting a 10GB file (using compression) with GPG took  6m34s (again reading from rust, writing to SSD). 

gpg with no compression takes 37s

 

While running following commands my help to find the bottleneck:

`vmstat -w 5`

or

`iostat`

to check I/O wait

`watch ps -p <put pid here> -L -o pid,tid,psr,pcpu,docker,args`

allow to see how the load is spread between threads and which CPU they running on. (keep in mind that 100% means full usage of all cores in your system. In your case 6.25% means the process is occupying a full core )

There's no I/O wait the whole time, the one CPU sits at 96-100% and the overall average 5 minute load is ~ 1.3 

 

Btw. are you running on bare metall or any virtualization?

The backed up server is bare metal

 

Bye

 Thomas 

Keep in mind, that python by default is not able to take advantage from multiple cores. Duplicity has some functionality to share load across processes to bypass this limitation, but this still have some boundaries. 

Yeah  -  I was hoping to be able to get at least a few parallel encryptions going and ideally

also a few parallel network upload streams (as aws cli does).

  

Cheers,

Tink