Re: [Duplicity-talk] frequent 'Connection reset by peer' errors

[Vladimir Marek]

On Thu, Mar 02, 2017 at 02:14:10PM +0100, edgar.soldin--- via Duplicity-talk 
wrote:
> On 02.03.2017 11:26, Vladimir Marek via Duplicity-talk wrote:
> > Hi,
> > 
> > First of all, thanks for duplicity!
> > 
> > I have compiled duplicity for the Turris Omnia router[1] which uses
> > OpenWRT fork. I am making backup of ~700G of my documents (photos, hg
> > repositories, documtes etc) to onedrive. All worked really well for past
> > 14 days or so when suddenly I started to receive many of these
> > tracebacks:
> > 
> > Traceback (most recent call last):
> >   File "/usr/dup/usr/bin/duplicity", line 1546, in <module>
> >     with_tempdir(main)
> >   File "/usr/dup/usr/bin/duplicity", line 1540, in with_tempdir
> >     fn()
> >   File "/usr/dup/usr/bin/duplicity", line 1375, in main
> >     action = commandline.ProcessCommandLine(sys.argv[1:])
> >   File "/usr/dup/usr/lib/python2.7/site-packages/duplicity/commandline.py", 
> > line 1126, in ProcessCommandLine
> >     backup, local_pathname = set_backend(args[0], args[1])
> >   File "/usr/dup/usr/lib/python2.7/site-packages/duplicity/commandline.py", 
> > line 1015, in set_backend
> >     globals.backend = backend.get_backend(bend)
> >   File "/usr/dup/usr/lib/python2.7/site-packages/duplicity/backend.py", 
> > line 223, in get_backend
> >     obj = get_backend_object(url_string)
> >   File "/usr/dup/usr/lib/python2.7/site-packages/duplicity/backend.py", 
> > line 209, in get_backend_object
> >     return factory(pu)
> >   File 
> > "/usr/dup/usr/lib/python2.7/site-packages/duplicity/backends/onedrivebackend.py",
> >  line 91, in __init__
> >     self.resolve_directory()
> >   File 
> > "/usr/dup/usr/lib/python2.7/site-packages/duplicity/backends/onedrivebackend.py",
> >  line 189, in resolve_directory
> >     files = self.get_files(object_id)
> >   File 
> > "/usr/dup/usr/lib/python2.7/site-packages/duplicity/backends/onedrivebackend.py",
> >  line 217, in get_files
> >     response = self.http_client.get(self.API_URI + path + '/files')
> >   File "/usr/dup/usr/lib/python2.7/site-packages/requests/sessions.py", 
> > line 501, in get
> >     return self.request('GET', url, **kwargs)
> >   File 
> > "/usr/dup/usr/lib/python2.7/site-packages/requests_oauthlib/oauth2_session.py",
> >  line 358, in request
> >     headers=headers, data=data, **kwargs)
> >   File "/usr/dup/usr/lib/python2.7/site-packages/requests/sessions.py", 
> > line 488, in request
> >     resp = self.send(prep, **send_kwargs)
> >   File "/usr/dup/usr/lib/python2.7/site-packages/requests/sessions.py", 
> > line 609, in send
> >     r = adapter.send(request, **kwargs)
> >   File "/usr/dup/usr/lib/python2.7/site-packages/requests/adapters.py", 
> > line 479, in send
> >     raise ConnectionError(err, request=request)
> > ConnectionError: ('Connection aborted.', error(104, 'Connection reset by 
> > peer'))
> > 
> > 
> > I'm not sure why, my network connection seems to be reliable at the same
> > time. The rate of the problem is so high that duplicity became virtually
> > unusable to me (I have to retry connection five times before it works
> > once). So I started to look in adapters.py and after short fiddling I
> > changed
> > 
> > --- /usr/dup/usr/lib/python2.7/site-packages/requests/adapters.py.orig  
> > 2017-03-02 11:15:12.782282496 +0100
> > +++ /usr/dup/usr/lib/python2.7/site-packages/requests/adapters.py       
> > 2017-03-02 11:15:23.003037590 +0100
> > @@ -423,7 +423,7 @@
> >                      assert_same_host=False,
> >                      preload_content=False,
> >                      decode_content=False,
> > -                    retries=self.max_retries,
> > +                    retries=20, # self.max_retries,
> >                      timeout=timeout
> >                  )
> > 
> > Out of pure luck that seems to help me and I don't see the problems
> > anymore. I know that this is not proper bug report and not real fix
> > (it's not even in duplicity code), but couldn't something to that effect
> > be added to duplicity?
> > 
> > The self.max_retries is 0, so no retry is performed. No matter what
> > --num-retries command line option says. Maybe there would be a way to
> > bind --num-retries option to be set here too?
> > 
> > Sorry I don't know python nor I know any of the libraries involved so I
> > don't try to pretend I know how the situation could be improved.
> > 
> > [1] https://omnia.turris.cz/en/
> > 
> > Thank you
> > 
> 
> hi Vladimir,
> 
> can you revert your patch and try instead to patch 
> "duplicity/backends/onedrivebackend.py" line 177 from
> 
> def resolve_directory(self):
> 
> to
> 
> @retry('resolve_directory', fatal=True)
> def resolve_directory(self):
> 
> and see if the it now respects the --num-retries parameter?

Hmm, not really.

I had to copy the function 'retry' from duplicity/backend.py to
onedrivebackend.py too otherwise I got some syntax error.

I'm using this print in requests/adapters.py

        print self.max_retries
        print timeout

And it says

Retry(total=0, connect=None, read=False, redirect=None)
Timeout(connect=None, read=None, total=None)

when running

duplicity -v 9 \
  --archive-dir=/tmp/run/mountd/sda1/duplicity_cache \
  --asynchronous-upload \
  --tempdir=/tmp/run/mountd/sda1/duplicity_tmp \
  --num-retries=765 \
  --dry-run \
  /tmp/run/mountd/sda1/Backup.new/Photos/2016 \
  onedrive://backup/Photos/2016

and later on third run

...
error(104, 'Connection reset by peer')


Thank you
-- 
        Vlad