Re: [Duplicity-talk] Amazon S3/socket error problems

[Mark Wardle]

To try and figure out what is going on with this, I have uploaded a
duplicity archive to S3 using the ruby s3sync program. I don't know
whether this will be helpful:

On my system, duplicity can successfully determine the
"collection-status" and "verify" a S3-based archive as one would
expect.

Once an archive is on S3, I have run an incremental backup to it. It
works - both if there are no new changes, and if new files are
created.



Mark




=================================

duplicity collection-status s3+http://wardle/Aperture
Connecting with backend: BotoBackend
Archive dir: None

Found 0 backup chains without signatures.

Found a complete backup chain with matching signature chain:
-------------------------
Chain start time: Sat Jan  5 15:26:26 2008
Chain end time: Sat Jan  5 15:26:26 2008
Number of contained backup sets: 1
Total number of contained volumes: 5597
 Type of backup set:                            Time:      Num volumes:
                Full         Sat Jan  5 15:26:26 2008              5597
-------------------------
No orphaned or incomplete backup sets found.


On 05/01/2008, Mark Wardle <address@hidden> wrote:
> Duplicity with S3 has never worked on my machine.
>
> I would be very happy to run a test program if it helps!
>
> Mark
>
> On 05/01/2008, Mitchell Garnaat <address@hidden> wrote:
> > There definitely isn't any inherent problem with boto/python/s3 and the Mac
> > platform because I developed boto on a Mac and use it everyday.  Does it
> > fail in this way consistently?  Has it ever worked?  Would you be willing to
> > run a small Python test program if I mailed it to you?
> >
> > Mitch
> >
> >
> > On Jan 5, 2008 1:44 PM, Mark Wardle <address@hidden> wrote:
> > > Dear Richard,
> > >
> > > Yes - I have the boto package installed.
> > >
> > > It was installed via macports, but just in case I have manually
> > > installed it from the .tar.gz file as well.
> > >
> > > Thanks,
> > >
> > > Mark
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On 05/01/2008, Richard Scott <address@hidden> wrote:
> > > > Hi,
> > > >
> > > > Do you have the Python Boto package installed?
> > > >
> > > > http://code.google.com/p/boto/
> > > >
> > > > Duplicity needs this to talk to Amazon S3.
> > > >
> > > > Cheers,
> > > >
> > > > Rich.
> > > >
> > > > > Dear all,
> > > > >
> > > > >
> > > > > I have been using duplicity successfully to backup (via scp/ssh) and I
> > > > > must say it is great - so thank you!
> > > > >
> > > > > However, I have started to look at Amazon S3. I have been using r3sync
> > > > > and jungledisk successfully, but have not had any success with
> > duplicity.
> > > > >
> > > > > My system is MacOSX, 10.4.11, with a current macports installation. In
> > > > > addition, I have upgraded the version of duplicity (manually) to the
> > latest version as I had these
> > > > > same problems with the previous version.
> > > > >
> > > > > duplicity @0.4.9_0 (active) librsync @0.9.7_1 (active) py-boto @1.0a_0
> > (active) py-pexpect @2.1_0
> > > > > (active)
> > > > > py-gnupg @ 0.3.2_1 (active) py25-boto @1.0a_0 (active) python24
> > @2.4.4_1+darwin_8 (active) python25
> > > > > @2.5.1_4+darwin_8 (active)
> > > > >
> > > > >
> > > > > I repeatedly get socket errors. I have reviewed the mailing list, and
> > > > > have seen the prior thread regarding this issue, but did not see a
> > solution posted.
> > > > >
> > > > > I have tested this using a cable modem internet connection as well as
> > > > > a university network (T1) with the same error. In addition, I have no
> > problem with other s3
> > > > > clients.
> > > > >
> > > > > I have not tried using boto directly as I don't know python but wonder
> > > > > whether this a specific Mac OS X problem as other users haven't
> > reported it?
> > > > >
> > > > > The transcript is shown below.
> > > > >
> > > > >
> > > > > Are there any workarounds possible, or can anyone suggest any
> > > > > tests/scripts to run to try and isolate the problem a little more?
> > > > >
> > > > > Alternatively, I could run a hybrid solution using duplicity to create
> > > > > an archive and then use s3sync to upload the files to S3 separately. I
> > presume there is no
> > > > > complication in moving an archive around in this way?
> > > > >
> > > > > Many thanks,
> > > > >
> > > > >
> > > > > Best wishes,
> > > > >
> > > > >
> > > > > Mark
> > > > >
> > > > >
> > > > > ================================================
> > > > >
> > > > >
> > > > >
> > > > > Daisy:~ mark$ duplicity -v9 /Users/mark/Repos s3+http://wardle
> > > > > Main action: inc
> > > > > Collection Status
> > > > > -----------------
> > > > > Connecting with backend: BotoBackend
> > > > > Archive dir: None
> > > > >
> > > > >
> > > > > Found 0 backup chains without signatures.
> > > > > No backup chains with active signatures found
> > > > > No orphaned or incomplete backup sets found.
> > > > > Last full backup date: Thu Jan  1 01:00:00 1970
> > > > > No signatures found, switching to full backup.
> > > > > Using temporary directory /tmp/duplicity-YkKddu-tempdir
> > > > > Registering (mktemp) temporary file
> > > > > /tmp/duplicity-YkKddu-tempdir/mktemp-Kzk1h6-1
> > > > > Registering (mktemp) temporary file
> > > > > /tmp/duplicity-YkKddu-tempdir/mktemp-Xb73NX-2
> > > > > Selecting /Users/mark/Repos
> > > > > Comparing () and None
> > > > > Getting delta of (() /Users/mark/Repos dir) and None
> > > > > Generating delta - new file: .
> > > > > Selecting /Users/mark/Repos/.DS_Store
> > > > > Comparing ('.DS_Store',) and None
> > > > > Getting delta of (('.DS_Store',) /Users/mark/Repos/.DS_Store reg) and
> > None
> > > > > Generating delta - new file: .DS_Store
> > > > > Selecting /Users/mark/Repos/Papers
> > > > > Comparing ('Papers',) and None
> > > > > Getting delta of (('Papers',) /Users/mark/Repos/Papers dir) and None
> > > > > Generating delta - new file: Papers
> > > > > Selecting /Users/mark/Repos/Papers/README.txt
> > > > > Comparing ('Papers', 'README.txt') and None
> > > > > Getting delta of (('Papers', ' README.txt')
> > > > > /Users/mark/Repos/Papers/README.txt reg) and None
> > > > > Generating delta - new file: Papers/README.txt
> > > > > Selecting /Users/mark/Repos/Papers/conf
> > > > > Comparing ('Papers', 'conf') and None
> > > > > Getting delta of (('Papers', 'conf') /Users/mark/Repos/Papers/conf
> > dir) and None
> > > > > Generating delta - new file: Papers/conf
> > > > > Selecting /Users/mark/Repos/Papers/conf/authz
> > > > > Comparing ('Papers', 'conf', 'authz') and None
> > > > > Getting delta of (('Papers', 'conf', 'authz')
> > > > > /Users/mark/Repos/Papers/conf/authz reg) and None
> > > > > Generating delta - new file: Papers/conf/authz
> > > > > Selecting /Users/mark/Repos/Papers/conf/passwd
> > > > > Comparing ('Papers', 'conf', 'passwd') and None
> > > > > Getting delta of (('Papers', 'conf', 'passwd')
> > > > > /Users/mark/Repos/Papers/conf/passwd reg) and None
> > > > > Generating delta - new file: Papers/conf/passwd
> > > > > Selecting /Users/mark/Repos/Papers/conf/svnserve.conf
> > > > > Comparing ('Papers', 'conf', ' svnserve.conf') and None
> > > > > Getting delta of (('Papers', 'conf', 'svnserve.conf')
> > > > > /Users/mark/Repos/Papers/conf/svnserve.conf reg) and
> > None
> > > > > Generating delta - new file: Papers/conf/svnserve.conf
> > > > > Selecting /Users/mark/Repos/Papers/dav
> > > > > Comparing ('Papers', 'dav') and None
> > > > > Getting delta of (('Papers', 'dav') /Users/mark/Repos/Papers/dav dir)
> > and None
> > > > >
> > > > >
> > > > > [...] cut
> > > > > Generating delta - new file: Papers/db/revs/0
> > > > > Selecting /Users/mark/Repos/Papers/db/revs/1
> > > > > Comparing ('Papers', 'db', 'revs', '1') and None
> > > > > Getting delta of (('Papers', 'db', 'revs', '1')
> > > > > /Users/mark/Repos/Papers/db/revs/1 reg) and None
> > > > > Generating delta - new file: Papers/db/revs/1
> > > > > Uploading s3+http://wardle/duplicity-
> > full.2008-01-05T11:19:48Z.vol1.difftar.gpg
> > > > > Caught a socket error, trying to recover
> > > > > Caught a socket error, trying to recover
> > > > > [...] cut   (endlessly loops until I CTRL-C - which interestingly
> > > > > always interrupts gpg rather than duplicity directly?) Caught a socket
> > error, trying to recover
> > > > > ^CUpload
> > 's3+http://wardle/duplicity-full.2008-01-05T11:19:48Z.vol1.difftar.gpg'
> > > > > failed (attempt #1)
> > > > >
> > > > > gpg: Interrupt caught ... exiting
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Dr. Mark Wardle
> > > > > Specialist registrar, Neurology
> > > > > Cardiff, UK
> > > > >
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Duplicity-talk mailing list
> > > > > address@hidden
> > > > >
> > http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Duplicity-talk mailing list
> > > > address@hidden
> > > >
> > http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> > > >
> > > >
> > > >
> > ______________________________________________________________________
> > > > This email has been scanned by the MessageLabs Email Security System.
> > > > For more information please visit
> > http://www.messagelabs.com/email
> > > >
> > ______________________________________________________________________
> > > >
> > >
> > >
> > > --
> > >
> > >
> > >
> > > Dr. Mark Wardle
> > > Specialist registrar, Neurology
> > > Cardiff, UK
> > >
> > >
> > > _______________________________________________
> > > Duplicity-talk mailing list
> > > address@hidden
> > >
> > http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> > >
> >
> >
> > ______________________________________________________________________
> >  This email has been scanned by the MessageLabs Email Security System.
> >  For more information please visit
> > http://www.messagelabs.com/email
> > ______________________________________________________________________
> >
> > _______________________________________________
> > Duplicity-talk mailing list
> > address@hidden
> > http://lists.nongnu.org/mailman/listinfo/duplicity-talk
> >
> >
>
>
> --
> Dr. Mark Wardle
> Specialist registrar, Neurology
> Cardiff, UK
>


-- 
Dr. Mark Wardle
Specialist registrar, Neurology
Cardiff, UK