Re: [Duplicity-talk] frequent 'Connection reset by peer' errors

[edgar . soldin]

On 02.03.2017 14:56, Vladimir Marek wrote:
> On Thu, Mar 02, 2017 at 02:14:10PM +0100, edgar.soldin--- via Duplicity-talk 
> wrote:
>> On 02.03.2017 11:26, Vladimir Marek via Duplicity-talk wrote:
>>> Hi,
>>>
>>> First of all, thanks for duplicity!
>>>
>>> I have compiled duplicity for the Turris Omnia router[1] which uses
>>> OpenWRT fork. I am making backup of ~700G of my documents (photos, hg
>>> repositories, documtes etc) to onedrive. All worked really well for past
>>> 14 days or so when suddenly I started to receive many of these
>>> tracebacks:
>>>
>>> Traceback (most recent call last):
>>>   File "/usr/dup/usr/bin/duplicity", line 1546, in <module>
>>>     with_tempdir(main)
>>>   File "/usr/dup/usr/bin/duplicity", line 1540, in with_tempdir
>>>     fn()
>>>   File "/usr/dup/usr/bin/duplicity", line 1375, in main
>>>     action = commandline.ProcessCommandLine(sys.argv[1:])
>>>   File "/usr/dup/usr/lib/python2.7/site-packages/duplicity/commandline.py", 
>>> line 1126, in ProcessCommandLine
>>>     backup, local_pathname = set_backend(args[0], args[1])
>>>   File "/usr/dup/usr/lib/python2.7/site-packages/duplicity/commandline.py", 
>>> line 1015, in set_backend
>>>     globals.backend = backend.get_backend(bend)
>>>   File "/usr/dup/usr/lib/python2.7/site-packages/duplicity/backend.py", 
>>> line 223, in get_backend
>>>     obj = get_backend_object(url_string)
>>>   File "/usr/dup/usr/lib/python2.7/site-packages/duplicity/backend.py", 
>>> line 209, in get_backend_object
>>>     return factory(pu)
>>>   File 
>>> "/usr/dup/usr/lib/python2.7/site-packages/duplicity/backends/onedrivebackend.py",
>>>  line 91, in __init__
>>>     self.resolve_directory()
>>>   File 
>>> "/usr/dup/usr/lib/python2.7/site-packages/duplicity/backends/onedrivebackend.py",
>>>  line 189, in resolve_directory
>>>     files = self.get_files(object_id)
>>>   File 
>>> "/usr/dup/usr/lib/python2.7/site-packages/duplicity/backends/onedrivebackend.py",
>>>  line 217, in get_files
>>>     response = self.http_client.get(self.API_URI + path + '/files')
>>>   File "/usr/dup/usr/lib/python2.7/site-packages/requests/sessions.py", 
>>> line 501, in get
>>>     return self.request('GET', url, **kwargs)
>>>   File 
>>> "/usr/dup/usr/lib/python2.7/site-packages/requests_oauthlib/oauth2_session.py",
>>>  line 358, in request
>>>     headers=headers, data=data, **kwargs)
>>>   File "/usr/dup/usr/lib/python2.7/site-packages/requests/sessions.py", 
>>> line 488, in request
>>>     resp = self.send(prep, **send_kwargs)
>>>   File "/usr/dup/usr/lib/python2.7/site-packages/requests/sessions.py", 
>>> line 609, in send
>>>     r = adapter.send(request, **kwargs)
>>>   File "/usr/dup/usr/lib/python2.7/site-packages/requests/adapters.py", 
>>> line 479, in send
>>>     raise ConnectionError(err, request=request)
>>> ConnectionError: ('Connection aborted.', error(104, 'Connection reset by 
>>> peer'))
>>>
>>>
>>> I'm not sure why, my network connection seems to be reliable at the same
>>> time. The rate of the problem is so high that duplicity became virtually
>>> unusable to me (I have to retry connection five times before it works
>>> once). So I started to look in adapters.py and after short fiddling I
>>> changed
>>>
>>> --- /usr/dup/usr/lib/python2.7/site-packages/requests/adapters.py.orig  
>>> 2017-03-02 11:15:12.782282496 +0100
>>> +++ /usr/dup/usr/lib/python2.7/site-packages/requests/adapters.py       
>>> 2017-03-02 11:15:23.003037590 +0100
>>> @@ -423,7 +423,7 @@
>>>                      assert_same_host=False,
>>>                      preload_content=False,
>>>                      decode_content=False,
>>> -                    retries=self.max_retries,
>>> +                    retries=20, # self.max_retries,
>>>                      timeout=timeout
>>>                  )
>>>
>>> Out of pure luck that seems to help me and I don't see the problems
>>> anymore. I know that this is not proper bug report and not real fix
>>> (it's not even in duplicity code), but couldn't something to that effect
>>> be added to duplicity?
>>>
>>> The self.max_retries is 0, so no retry is performed. No matter what
>>> --num-retries command line option says. Maybe there would be a way to
>>> bind --num-retries option to be set here too?
>>>
>>> Sorry I don't know python nor I know any of the libraries involved so I
>>> don't try to pretend I know how the situation could be improved.
>>>
>>> [1] https://omnia.turris.cz/en/
>>>
>>> Thank you
>>>
>>
>> hi Vladimir,
>>
>> can you revert your patch and try instead to patch 
>> "duplicity/backends/onedrivebackend.py" line 177 from
>>
>> def resolve_directory(self):
>>
>> to
>>
>> @retry('resolve_directory', fatal=True)
>> def resolve_directory(self):
>>
>> and see if the it now respects the --num-retries parameter?
> 
> Hmm, not really.
> 
> I had to copy the function 'retry' from duplicity/backend.py to
> onedrivebackend.py too otherwise I got some syntax error.

yeah right, sorry, forgot the import statement. add on top after all the other 
imports

from duplicity.backend import retry

> I'm using this print in requests/adapters.py
> 
>       print self.max_retries
>       print timeout
> 
> And it says
> 
> Retry(total=0, connect=None, read=False, redirect=None)
> Timeout(connect=None, read=None, total=None)

well, that won't change. but what will change is that duplicity itself will 
retry on failure. i had a short look on how to configure the retry in urllib3 
but that seemed cumbersome.

> when running
> 
> duplicity -v 9 \
>   --archive-dir=/tmp/run/mountd/sda1/duplicity_cache \
>   --asynchronous-upload \
>   --tempdir=/tmp/run/mountd/sda1/duplicity_tmp \
>   --num-retries=765 \
>   --dry-run \
>   /tmp/run/mountd/sda1/Backup.new/Photos/2016 \
>   onedrive://backup/Photos/2016
> 
> and later on third run
> 
> ...
> error(104, 'Connection reset by peer')
> 

applicating the retry decorator you should see the error but, but also the 
retries up to --num-retries count or success.

..ede/duply.net