Re: [Duplicity-talk] Backend for Amazon S3

[Kenneth Loafman]

I'm working on the RC9 patches at the moment.  This looks a bit bare
bones.  Don't we need to address retries for network errors, timeouts,
etc., or is that handled in boto itself?

...Thanks,
...Ken

Eric Evans wrote:
> I filed a bug[1] for this a couple of days ago, but thought it might 
> warrant a post here in case others are interested in testing.
> 
> The current duplicity backend for S3 uses the bitbucket module from 
> Mitch Garnaat which has been deprecated in favor of boto[2].
> 
> I've worked up a patch for a boto-using-backend which is attached to the
> bug report and this email. Use of this backend is identical to the
> current one with one exception, you need to export the AWS_ACCESS_KEY_ID 
> and AWS_SECRET_ACCESS_KEY environment variables with your Amazon key id 
> and secret as opposed to the current S3KEY/S3SECRET combo.
> 
> I'm also working to get boto into the Debian archive and currently have a
> package waiting in the NEW queue. Any Debian users interested in testing
> this patch can grab the package using the following sources.
> 
> deb http://people.debian.org/~eevans/python-boto binary/
> deb-src http://people.debian.org/~eevans/python-boto source/
> 
> Any feedback would be much appreciated.
> 
> Regards,
> 
> 
> [1] http://savannah.nongnu.org/bugs/?20339
> [2] http://code.google.com/p/boto/
> 
> 
> 
> ------------------------------------------------------------------------
> 
> --- a/duplicity/backends.py   Fri Jun 29 20:09:26 2007 +0000
> +++ b/duplicity/backends.py   Tue Jul 03 16:27:13 2007 -0500
> @@ -680,6 +680,62 @@ class BitBucketBackend(Backend):
>                                                                  % file)
>                               self._connect()
>                               del self.bucket[file]
> +
> +class BotoBackend(Backend):
> +     """
> +     Backend for Amazon's Simple Storage System, (aka Amazon S3), though
> +     the use of the boto module, (http://code.google.com/p/boto/).
> +
> +     To make use of this backend you must export the environment variables
> +     AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with your Amazon Web 
> +     Services key id and secret respectively.
> +     """
> +
> +     def __init__(self, parsed_url):
> +             try:
> +                     from boto.s3.connection import S3Connection
> +                     from boto.s3.key import Key
> +             except ImportError:
> +                     raise BackendException("This backend requires the boto 
> library, " \
> +                             "(http://code.google.com/p/boto/).")
> +
> +             self.key_class = Key
> +
> +             self.conn = S3Connection()
> +             self.bucket_name = parsed_url.suffix
> +             self.bucket = self.conn.create_bucket(self.bucket_name)
> +
> +             if not (os.environ.has_key('AWS_ACCESS_KEY_ID') and 
> +                             os.environ.has_key('AWS_SECRET_ACCESS_KEY')):
> +                     raise BackendException("The AWS_ACCESS_KEY_ID and " \
> +                             "AWS_SECRET_ACCESS_KEY environment variables 
> are not set.")
> +
> +             if '/' in self.bucket_name:
> +                     raise BackendException("Invalid bucket specification.")
> +
> +     def put(self, source_path, remote_filename=None):
> +             if not remote_filename:
> +                     remote_filename = source_path.get_filename()
> +             log.Log("Uploading %s to Amazon S3" % remote_filename, 5)
> +             key = self.key_class(self.bucket)
> +             key.key = remote_filename
> +             key.set_contents_from_filename(source_path.name, 
> +                             {'Content-Type': 'application/octet-stream'})
> +     
> +     def get(self, remote_filename, local_path):
> +             log.Log("Downloading %s from Amazon S3" % remote_filename, 5)
> +             key = self.key_class(self.bucket)
> +             key.key = remote_filename
> +             key.get_contents_to_filename(local_path.name)
> +             local_path.setdata()
> +
> +     def list(self):
> +             filename_list = [k.key for k in self.bucket.get_all_keys()]
> +             return filename_list
> +
> +     def delete(self, filename_list):
> +             for filename in filename_list:
> +                     self.bucket.delete_key(filename)
>  
>  class webdavBackend(Backend):
>       """Backend for accessing a WebDAV repository.
> @@ -838,5 +894,5 @@ protocol_class_dict = {"scp": scpBackend
>                                          "ftp": ftpBackend,
>                                          "hsi": hsiBackend,
>                                          "rsync": rsyncBackend,
> -                                        "s3+http": BitBucketBackend,
> +                                        "s3+http": BotoBackend,
>                                          "webdav": webdavBackend}
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk