[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upl
From: |
Kris |
Subject: |
Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http:// |
Date: |
Wed, 2 Nov 2011 21:30:05 -0400 |
Thanks for your help Michael! It's hugely appreciated. I can confirm that I was
looking at the same code and saw that the last chunk of a file not perfectly
divisible will not have an incorrect number of bytes passed in to param for the
last chunk.
Just to confirm, I believe the value passed to the 'bytes' key should be:
bytes - (chunk_size*(chunk_size-1))
Kris
On 2011-11-02, at 9:18 PM, Michael Terry wrote:
> On 2 November 2011 19:34, Kris <address@hidden> wrote:
>> I'm taking a look at it again right now. Just a FYI before I dig in, I
>> removed the volsize cmd line and it did the same thing. The default volsize
>> is 25MB in duplicity and that's what the tmp file is. The remote file
>> however is 50 megs. The PoolWorkers split the chunk in to two 25 meg chunks
>> and uploaded those, which isn't proper.
>
> Good debugging work, Kris. This isn't bug 498933 after all, but is
> instead a bug in 0.6.16 specifically. In that version, the S3 backend
> got new 'multichunk' support.
>
> But it seems to have a bug that when the size of the volume isn't
> perfectly divisible by the multichunk size, it will upload too much
> data. I've filed bug https://bugs.launchpad.net/duplicity/+bug/885513
> about it.
>
> For now, I believe you can workaround it with
> --s3-multipart-chunk-size=1000 (causing duplicity to basically ignore
> the multipart code).
>
> I feel like suggesting that maybe the multipart branch should be
> backed out until it can be stress tested a bit more? (so far we've
> noticed this bug and bug 878220 (the one where botobackend.py added a
> new, uncommon, and unconditional import due to the patch)
>
> -mt
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, (continued)
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Michael Terry, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Michael Terry, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Michael Terry, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, edgar . soldin, 2011/11/02
- Message not available
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, edgar . soldin, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, edgar . soldin, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Michael Terry, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://,
Kris <=
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Michael Terry, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/02
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, edgar . soldin, 2011/11/03
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/03
- Re: [Duplicity-talk] Duplicity 0.6.16 "File XXX was corrupted during upload." to s3+http://, Kris, 2011/11/03