Re: [Duplicity-talk] new ftp backend

[Kenneth Loafman]

Thanks for the work.  I've been thinking of going this route for a while
and a recent attempt at fixing Python's ftplib via patch submission just
sealed the deal.  The folks at ncftp at least realize that not all FTP
servers are RFC compliant and make amends for that.  The goal is a
working backup, not enforcing RFC compliance on everyone.

Since ncftp is supported on both Windows and Linux, I have no problems
with adopting it for our use.  I've used ncftp for years and have yet to
find a server it fails to work on.

I'll do a bit of testing myself, then get this into RC9, probably by the
end of the week.

...Ken

Thorsten Schnebeck wrote:
> Hi,
> 
> I tried to use the ftplicity/duplicity tool combo to backup my root 
> server to the ftp-only backup server of my host.
> 
> duplicity worked like charm - but only on single branches of my file tree. 
> Everytime I tried to backup the whole tree I got the strange connections 
> timeouts and socket errors.
> 
> I tried different versions, different patches from different places but I got 
> no stable backup processing.
> 
> When reading about all the problems that seems to be a combination of server, 
> timing and pythons ftplib *me* wonders that I did not have all these problems 
> with my old simple backup that use the scriptable ftp-client ncftp.
> 
> So, last week I deleted the current ftp-backend and wrote a new one based on 
> ncftp - and all problems are gone :-)
> 
> I can not say my backend is better than the current one, its more a "works 
> for 
> me" thing and I want to share the code, so it is not lost and can help 
> others. You need the attached patch (vs. duplicity-0.4.3.RC8) and of course 
> the ncftp package. I use
> 
> net-ftp/ncftp
>       Latest version installed: 3.1.9
>       Homepage:      http://www.ncftp.com/
> 
> I'm not really a python programmer, the code is adapted mainly from the rsync 
> backend code.
> 
> HTH
> 
>   Thorsten 
> 
> P.S. Yes, the delete function is a bit of a hack ;-)
> 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> --- src/backends.py.vanilla   2007-07-03 18:57:33.000000000 +0200
> +++ src/backends.py   2007-07-03 18:57:27.000000000 +0200
> @@ -18,7 +18,7 @@
>  
>  """Provides functions and classes for getting/sending files to destination"""
>  
> -import os, socket, types, ftplib, tempfile, time, sys
> +import os, socket, types, tempfile, time, sys
>  import log, path, dup_temp, file_naming
>  import base64, getpass, xml.dom.minidom, httplib, urllib
>  import socket
> @@ -335,170 +335,48 @@
>       """This backend uses sftp to perform file operations"""
>       pass # Do this later
>  
> -# TODO: handle login failures, by not retrying, but printing a helpful 
> error?!
> -#    e.g. """Caught exception <class 'duplicity.ftplib.error_perm'>: 521 
> unknown user (proxy auth failed) (#3), sleeping 20s before retry.."""
> -#       and  """Caught exception <class 'duplicity.ftplib.error_perm'>: 530 
> Login incorrect. (#2), sleeping 10s before retry.."""
> -class ftpBackend(Backend):
> -     """Connect to remote store using File Transfer Protocol"""
> -     RETRY_SLEEP = 10 # time in seconds before reconnecting on errors (gets 
> multiplied with the try counter)
> -     RETRIES = 15 # number of retries
> -
> -     def __init__(self, parsed_url):
> -             """Create a new ftp backend object, log in to host"""
> -             self.parsed_url = parsed_url
> -             self.ftp = False
> -             self.connect()
> -
> -     def connect(self):
> -             """Connect to self.parsed_url"""
> -             tries = 0
> -             while( True ):
> -                     tries = tries + 1
> -
> -                     if self.ftp:
> -                             self.close()
> -                     self.ftp = ftplib.FTP()
> -                     if log.verbosity > 8:
> -                             self.ftp.set_debuglevel(2)
> -                     try:
> -                             if self.parsed_url.port is None:
> -                                     self.log_wrap('connect', 
> self.parsed_url.host)
> -                             else:
> -                                     self.log_wrap('connect', 
> self.parsed_url.host,
> -                                             self.parsed_url.port)
> -
> -                             if self.parsed_url.user is not None:
> -                                     self.log_wrap('login', 
> self.parsed_url.user,
> -                                             self.get_password())
> -                             else: self.log_wrap('login')
> -
> -                             try:
> -                                     self.log_wrap('cwd', 
> self.parsed_url.path)
> -                             except ftplib.error_perm, e:
> -                                     if "550" in str(e):
> -                                             self.log_wrap('mkd', 
> self.parsed_url.path)
> -                                             self.log_wrap('cwd', 
> self.parsed_url.path)
> -                                     else: raise
> -                             # OK:
> -                             break
> -                     except ftplib.all_errors, e:
> -                             if tries > self.RETRIES:
> -                                     raise( BackendException(e) )
> -
> -                     sleep_time = self.RETRY_SLEEP * (tries-1)
> -                     import traceback
> -                     log.Log("Caught exception %s during connect: %s (#%d), 
> sleeping %ds before retry..\nTraceback:\n%s" % (
> -                                     sys.exc_info()[0],
> -                                     sys.exc_info()[1],
> -                                     tries,
> -                                     sleep_time,
> -                                     "".join(traceback.format_stack())), 7)
> -                     time.sleep(sleep_time)
> -
> -     def error_wrap(self, command, *args):
> -             """Run self.ftp.command(*args), but raise BackendException on 
> error"""
> -
> -             # Execute:
> -             tries = 0
> -             while( True ):
> -                     tries = tries+1
> -
> -                     log.Log("FTP: %s %s" % (command,args), 5)
> -
> -                     # Execute command:
> -                     try:
> -                             return ftplib.FTP.__dict__[command](self.ftp, 
> *args)
> -                     except ftplib.all_errors, e:
> -                             # 450 or 550 on list isn't an error, but 
> indicates an empty dir
> -                             # 104 indicates a reset connection, sometimes 
> instead of 450/550
> -                             if command is "nlst" and ("450" in str(e) or 
> "550" in str(e) or "104" in str(e)):
> -                                     return []
> -
> -                             # Give up, maybe
> -                             if tries > self.RETRIES:
> -                                     import traceback
> -                                     log.Warn("Caught exception %s: %s (%d 
> exceptions in total), giving up..\nTraceback:\n%s" % (
> -                                             sys.exc_info()[0],
> -                                             sys.exc_info()[1],
> -                                             tries,
> -                                             
> "".join(traceback.format_stack())
> -                                             ))
> -                                     raise BackendException(e)
> -
> -                             # Sleep and retry (after trying to reconnect, 
> if possible):
> -                             sleep_time = self.RETRY_SLEEP * (tries-1);
> -                             import traceback
> -                             log.Log("Caught exception %s: %s (#%d), 
> sleeping %ds before retry..\nTraceback:\n%s" % (
> -                                             sys.exc_info()[0],
> -                                             sys.exc_info()[1],
> -                                             tries,
> -                                             sleep_time,
> -                                             
> "".join(traceback.format_stack())), 9)
> -                             time.sleep(sleep_time)
> -
> -                             log.Log("Re-connecting...", 5)
> -                             self.connect()
> -                     else: break
> -
> -     def log_wrap(self, command, *args):
> -             """Log FTP command and execute it"""
> -             if command is 'login':
> -                     if log.verbosity > 8:
> -                             # Log full args at level 9:
> -                             log.Log("FTP: %s %s" % (command,args), 9)
> -                     else:
> -                             # replace password with stars:
> -                             log_args = list(args)
> -                             log_args[1] = '*' * len(log_args[1])
> -                             log.Log("FTP: %s %s" % (command,log_args), 5)
> -             else:
> -                     log.Log("FTP: %s %s" % (command,args), 5)
> -             return ftplib.FTP.__dict__[command](self.ftp, *args)
> -
> -     def get_password(self):
> -             """Get ftp password using environment if possible"""
> -             try: return os.environ['FTP_PASSWORD']
> -             except KeyError:
> -                     return getpass.getpass('Password for '+ 
> self.parsed_url.user + '@' + self.parsed_url.host + ': ')
> -
> -     def put(self, source_path, remote_filename = None):
> -             """Transfer source_path to remote_filename"""
> -             if not remote_filename: remote_filename = 
> source_path.get_filename()
> -             source_file = source_path.open("rb")
> -             log.Log("Saving %s on FTP server" % (remote_filename,), 5)
> -             self.error_wrap('storbinary', "STOR "+remote_filename, 
> source_file)
> -             assert not source_file.close()
>  
> -     def get(self, remote_filename, local_path):
> -             """Get remote filename, saving it to local_path"""
> -             target_file = local_path.open("wb")
> -             log.Log("Retrieving %s from FTP server" % (remote_filename,), 5)
> -             self.error_wrap('retrbinary', "RETR "+remote_filename,
> -                                             target_file.write)
> -             assert not target_file.close()
> -             local_path.setdata()
> -
> -     def list(self):
> -             """List files in directory"""
> -             log.Log("Listing files on FTP server", 5)
> -             try: return self.error_wrap('nlst')
> -             except BackendException, e:
> -                     if "425" in str(e):
> -                             log.Log("Turning passive mode OFF", 5)
> -                             self.ftp.set_pasv(False)
> -                             return self.list()
> -                     raise
> -
> -     def delete(self, filename_list):
> -             """Delete files in filename_list"""
> -             for filename in filename_list:
> -                     log.Log("Deleting %s from FTP server" % (filename,), 5)
> -                     self.error_wrap('delete', filename)
> -
> -     def close(self):
> -             """Shut down connection"""
> -             try: self.ftp.quit()
> -             except ftplib.all_errors: pass
> +class ftpBackend(Backend):
> +        """Connect to remote store using File Transfer Protocol"""
> +        def __init__(self, parsed_url):
> +                self.url_string = parsed_url.url_string
> +                if self.url_string[-1] != '/':
> +                        self.url_string += '/'
> +
> +        def get_password(self):
> +                """Get ftp password using environment if possible"""
> +                try: return os.environ['FTP_PASSWORD']
> +                except KeyError:
> +                        log.Log("FTP_PASSWORD not set, using empty ftp 
> password", 3)
> +                        return ''
> +
> +        def put(self, source_path, remote_filename = None):
> +                """Transfer source_path to remote_filename"""
> +                if not remote_filename: remote_filename = 
> source_path.get_filename()
> +                pu = ParsedUrl(self.url_string)
> +                remote_path = os.path.join (pu.path, 
> remote_filename).rstrip('\n')
> +                commandline = "ncftpput -V -u \'%s\' -p \'%s\' -c \'%s\' 
> \'%s\' < \'%s\'" % (pu.user, self.get_password(), pu.host, remote_path, 
> source_path.name)
> +                self.run_command(commandline)
> +
> +        def get(self, remote_filename, local_path):
> +                """Get remote filename, saving it to local_path"""
> +                remote_path = os.path.join (self.url_string, 
> remote_filename).rstrip('\n')
> +                commandline = "ncftpget -V -p \'%s\' -c \'%s\' > \'%s\'" % 
> (self.get_password(), remote_path, local_path.name)
> +                self.run_command(commandline)
> +                local_path.setdata()
> +
> +        def list(self):
> +                """List files in directory"""
> +                commandline = "ncftpls -1 -p \'%s\' \'%s\'" % 
> (self.get_password(), self.url_string)
> +                l = self.popen(commandline).split('\n')
> +                return filter(lambda x: x, l)
> +
> +        def delete(self, filename_list):
> +                """Delete files in filename_list"""
> +                pu = ParsedUrl(self.url_string)
> +                for filename in filename_list:
> +                        commandline = "ncftpls -1 -p \'%s\' -X \'DELE 
> /%s%s\' \'%s\' > /dev/null" % (self.get_password(), pu.path, filename, 
> self.url_string)
> +                        self.run_command(commandline)
>  
>  
>  class rsyncBackend(Backend):
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Duplicity-talk mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/duplicity-talk