Re: [Duplicity-talk] Do backup to or restore from S3 get slower as a bucket increases in size?

[Peter Schuller]

> I've been working on an upgrade to our backup procedure, and am
> considering creating a new sub-folder under an S3 bucket every time we
> do a full backup. It looks like from our logs that backups may be
> getting slower as a bucket gets more backup tar files, but I'm having
> trouble testing this theory (don't want to play around too much with
> our repository).

I don't think you should have to worry about S3 itself slowing down
with large buckets (I've not noticed that with buckets that are very
very large with many many files). However, given that you say you
experience a slowdown my first thought is - is this perhaps due to the
time it takes to list files? Iteration over all files in a bucket will
definitely be O(n) in the number of files. If you're accumulating a
*lot* of small files, that would be expected to be noticable. But then
that depends on what kind of absolute numbers you're seeing; if you're
backing up really large amounts of data maybe the extra time taken to
iterate over the items in the bucket is insignificant.

If individual put:s and get:s of files are indeed slower as the bucket
grows larger, I'd consider that a bug. If you have such evidence I
encourage you to contact Amazon support about it ;)

-- 
/ Peter Schuller