Re: [Duplicity-talk] async_concurrency > 1?

[Peter Schuller]

> I'm no python expert, but I poked though the source and I noticed
> the async_concurrency global, which hints that it could be set to more than
> 1 for multiple simultaneous uploads. I'm wondering if there are any
> obvious/known issues with upping that to something like 10? I
> found 
> http://lists.nongnu.org/archive/html/duplicity-talk/2010-08/msg00011.html
> which indicates that there are plans to extend that part of the code
> further, but I'm wondering if in the meantime I could just hack the source
> to set async_concurrency to 10 or whatever if --asynchronous-upload was
> specified.

The status is roughly what I said in the message you link to above, at
least so far as I know. I have not had the time to follow developments
closely or develop on duplicity myself in quite a while, so I haven't
followed through with that work.

For S3, I can confirm that boto is okay to use in a multi-threaded
environment. I don't recall whether there was something about the
duplicity S3 backend that meant it was not, but I don't *think* so. I
do think I suspected some of the other backends to not be concurrency
safe though.

In any case, before experimenting, consider:

(1) This is not supported on restore yet AFAIK; if you rely on the
performance improvement to that extent for backups, can you afford to
wait on a restore?

(2) Keep in mind that bugs can be subtle if they exist, such as
potentially causing corruption or incorrect backup data. Take care in
experimenting. Just because it seems to work at first, doesn't mean it
works correctly in general.

In other words, I advise caution :)

-- 
/ Peter Schuller