Re: [Duplicity-talk] S3 connection timeout

[Peter Schuller]

> I used Duplicity for quite some time with the old bitbucket backend,
> mostly succesfully.  Since that was switched out with boto, I've never
> had success doing a backup to S3.  On multiple OS X (10.4/10.5)
> machines, I get "caught a socket error, trying to recover" messages
> from s3/key.py (as you note).

Right now (or right "a while ago when I did the testing") my
particular problem was not limited to duplicity+boto. I tried a simple
file store using the aws-s3 Ruby library, and suffered a similar
problem:

Errno::EPIPE: Broken pipe
        from /usr/local/lib/ruby/1.8/net/protocol.rb:175:in `write'
        from /usr/local/lib/ruby/1.8/net/protocol.rb:175:in `write0'
        from /usr/local/lib/ruby/1.8/net/protocol.rb:151:in `write'
        from /usr/local/lib/ruby/1.8/net/protocol.rb:166:in `writing'
        from /usr/local/lib/ruby/1.8/net/protocol.rb:150:in `write'
        from 
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/bin/../lib/aws/s3/extensions.rb:291:in
 `send_request_with_body_stream'
        from /usr/local/lib/ruby/1.8/net/http.rb:1524:in `exec'
        from /usr/local/lib/ruby/1.8/net/http.rb:1045:in `request'
        from 
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/bin/../lib/aws/s3/connection.rb:43:in
 `request'
        from 
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/bin/../lib/aws/s3/connection.rb:48:in
 `call'
        from 
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/bin/../lib/aws/s3/connection.rb:48:in
 `request'
        from 
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/bin/../lib/aws/s3/base.rb:69:in 
`request'
        from 
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/bin/../lib/aws/s3/base.rb:83:in 
`put'
        from 
/usr/local/lib/ruby/gems/1.8/gems/aws-s3-0.4.0/bin/../lib/aws/s3/object.rb:239:in
 `store'
        from (irb):43
        from :0

This was consistent across two consecutive attempts. Stack trace isn't
really relevent; the remote end is closing the connection, period. The
question is why.

(As an side - does anyone have a feel for what the best avenue would
be to get some kind of feedback from Amazon people? Such as getting
somebody to check a log file or whatever against a tech support
fee. Would surely help since it feels likely that the s3 server is
closing the connection for some very specific reason.)

So whatever is the reason why I am consistently having huge trouble
right now, seems to be more than just a duplicity/boto issue.

That said, I *have*, even when s3 has worked generally well, seen
these socket errors issues fairly often. Volume uploads would
frequently have to be re-tried once or twice, but would eventually
succeed. This behavior of just *never* suceeding is new, and hopefully
temporary.

-- 
/ Peter Schuller

PGP userID: 0xE9758B7D or 'Peter Schuller <address@hidden>'
Key retrieval: Send an E-Mail to address@hidden
E-Mail: address@hidden Web: http://www.scode.org

pgpdhd6xysEiH.pgp
Description: PGP signature