[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[Duplicity-tracker] [patch #6730] Fix timing out for SSH backend
From: |
Michael Terry |
Subject: |
[Duplicity-tracker] [patch #6730] Fix timing out for SSH backend |
Date: |
Sun, 25 Jan 2009 14:55:02 +0000 |
User-agent: |
Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.5) Gecko/2008121622 Ubuntu/8.10 (intrepid) Firefox/3.0.5 |
URL:
<http://savannah.nongnu.org/patch/?6730>
Summary: Fix timing out for SSH backend
Project: duplicity
Submitted by: mikix
Submitted on: Sun 25 Jan 2009 09:55:00 AM EST
Category: None
Priority: 5 - Normal
Status: None
Privacy: Public
Assigned to: None
Originator Email:
Open/Closed: Open
Discussion Lock: Any
_______________________________________________________
Details:
There have been several reports of timeouts when there are many, many files
on the backend, for both ftp and ssh backends. This patch tries to solve it
for the ssh backend.
There was talk of trying to solve it in a generic way (either use wildcards
when listing files or store files in a hierarchical manner so individual
folder listings aren't large). But I think I found a bug in how the ssh
backend handles timeouts that is the real issue.
So, the ssh backend uses pexpect. Particularly, it use's pexpect's idea of a
timeout. When in fact, it should use ssh's idea.
pexpect will timeout when it hasn't received any 'match' from the command in
X number of seconds. But for a large "ls -1" listing, that timeout can easily
pass while the "ls" is thinking or passing the sometimes large list of files
back (over several megs for huge listings).
What it should do instead is use ssh's idea of a timeout, which will actually
test whether the connection is still alive. This is controlled by the
ServerAliveInterval option.
This method does introduce a possible bug: if either ssh or the command it
runs (ls, mkdir, cd) hangs, we won't detect it. We'd hang too. I feel like
the chances of that are low (ls and such are very well tested), and in any
case, it's impossible to differentiate between a hung ls and an ls that is
taking a long time. The famous halting problem. :)
We could hedge our bets around that by also having some pexpect timeout that
is very large, like a couple minutes... But honestly, I don't think it's
needed. And without such a hedge, we can handle an infinitely large number of
files on the backend. In theory. :)
This patch does a few things:
1) Adds the ServerAliveInterval flag to ssh_options
2) Changes pexpect calls to timeout based on ssh message, not expect.TIMEOUT
3) Uses the trick from
http://lists.gnu.org/archive/html/duplicity-talk/2008-12/msg00030.html which
makes parsing the ls command much faster. This isn't directly related to the
above fix, but it's a nice performance enhancer while I was in there.
4) Fixes the return value from an sftp command to not include the echo of the
sftp command as the first listing. This meant that previously, the ssh
backend always thought there was a file called " ls -1" on the server. Not
that it matters, because it isn't a duplicity file, but I was there, so I
fixed it.
_______________________________________________________
File Attachments:
-------------------------------------------------------
Date: Sun 25 Jan 2009 09:55:00 AM EST Name: ssh-timeout.diff Size: 6kB
By: mikix
<http://savannah.nongnu.org/patch/download.php?file_id=17322>
_______________________________________________________
Reply to this item at:
<http://savannah.nongnu.org/patch/?6730>
_______________________________________________
Message sent via/by Savannah
http://savannah.nongnu.org/
- [Duplicity-tracker] [patch #6730] Fix timing out for SSH backend,
Michael Terry <=