duplicity-tracker
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Duplicity-tracker] [patch #6730] Fix timing out for SSH backend


From: Michael Terry
Subject: [Duplicity-tracker] [patch #6730] Fix timing out for SSH backend
Date: Sun, 25 Jan 2009 14:55:02 +0000
User-agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.5) Gecko/2008121622 Ubuntu/8.10 (intrepid) Firefox/3.0.5

URL:
  <http://savannah.nongnu.org/patch/?6730>

                 Summary: Fix timing out for SSH backend
                 Project: duplicity
            Submitted by: mikix
            Submitted on: Sun 25 Jan 2009 09:55:00 AM EST
                Category: None
                Priority: 5 - Normal
                  Status: None
                 Privacy: Public
             Assigned to: None
        Originator Email: 
             Open/Closed: Open
         Discussion Lock: Any

    _______________________________________________________

Details:

There have been several reports of timeouts when there are many, many files
on the backend, for both ftp and ssh backends.  This patch tries to solve it
for the ssh backend.

There was talk of trying to solve it in a generic way (either use wildcards
when listing files or store files in a hierarchical manner so individual
folder listings aren't large).  But I think I found a bug in how the ssh
backend handles timeouts that is the real issue.

So, the ssh backend uses pexpect.  Particularly, it use's pexpect's idea of a
timeout.  When in fact, it should use ssh's idea.

pexpect will timeout when it hasn't received any 'match' from the command in
X number of seconds.  But for a large "ls -1" listing, that timeout can easily
pass while the "ls" is thinking or passing the sometimes large list of files
back (over several megs for huge listings).

What it should do instead is use ssh's idea of a timeout, which will actually
test whether the connection is still alive.  This is controlled by the
ServerAliveInterval option.

This method does introduce a possible bug: if either ssh or the command it
runs (ls, mkdir, cd) hangs, we won't detect it.  We'd hang too.  I feel like
the chances of that are low (ls and such are very well tested), and in any
case, it's impossible to differentiate between a hung ls and an ls that is
taking a long time.  The famous halting problem.  :)

We could hedge our bets around that by also having some pexpect timeout that
is very large, like a couple minutes...  But honestly, I don't think it's
needed.  And without such a hedge, we can handle an infinitely large number of
files on the backend.  In theory.  :)

This patch does a few things:
1) Adds the ServerAliveInterval flag to ssh_options

2) Changes pexpect calls to timeout based on ssh message, not expect.TIMEOUT

3) Uses the trick from
http://lists.gnu.org/archive/html/duplicity-talk/2008-12/msg00030.html which
makes parsing the ls command much faster.  This isn't directly related to the
above fix, but it's a nice performance enhancer while I was in there.

4) Fixes the return value from an sftp command to not include the echo of the
sftp command as the first listing.  This meant that previously, the ssh
backend always thought there was a file called " ls -1" on the server.  Not
that it matters, because it isn't a duplicity file, but I was there, so I
fixed it.



    _______________________________________________________

File Attachments:


-------------------------------------------------------
Date: Sun 25 Jan 2009 09:55:00 AM EST  Name: ssh-timeout.diff  Size: 6kB  
By: mikix

<http://savannah.nongnu.org/patch/download.php?file_id=17322>

    _______________________________________________________

Reply to this item at:

  <http://savannah.nongnu.org/patch/?6730>

_______________________________________________
  Message sent via/by Savannah
  http://savannah.nongnu.org/





reply via email to

[Prev in Thread] Current Thread [Next in Thread]