Re: [rdiff-backup-users] Re: Remote encrypted backup with slow connection.

[Dominic Raferd]

Piotr Karbowski wrote:
> On Mon, Sep 21, 2009 at 8:21 PM, Dominic Raferd <address@hidden> wrote:
>   
> > Piotr Karbowski wrote:
> >     
> > > On Mon, Sep 21, 2009 at 2:02 PM, Matthew Miller <address@hidden> wrote:
> > >
> > >       
> > > > On Mon, Sep 21, 2009 at 01:58:11PM +0200, Piotr Karbowski wrote:
> > > >
> > > >         
> > > > > local rdiff-backup dir with remote server but how? If I will use for
> > > > > example rsync it still need to check whole files for changes (read,
> > > > > download it) and upload only new. I hope you will understand what I
> > > > > need and help me.
> > > > >
> > > > >           
> > > > rsync won't check whole files unless you give the -c flag. Otherwise, it
> > > > just compares metadata. I don't know if that's also the case with
> > > > rdiff-backup, but I assume so.
> > > >
> > > >         
> > > So I need to know how rdiff default compares data, if by size and
> > > mod-time, it will not be so painful but still itefficient  will download changed
> > > files to generate diff.
> > >       
> > Rdiff-backup is designed to be ultra-efficient at this activity. It only
> > sends the changes in a file over the wire, not the whole file. To do this it
> > uses the librsync library which is effectively the same as rsync. You can
> > read more about the technique at http://en.wikipedia.org/wiki/Rsync.
> > rdiff-backup does not use file times to determine whether to do backups. It
> > can backup very large files with small changes very quickly.
> >
> > Dominic
> >
> >     
>
> You dont understand me, rdiff-backup is efficient, but to make diff it
> must read WHOLE file, on remote nfs or sshfs it is SLOOOOW and
> painful
Sorry I get it now. But I think rdiff-backup and rsync require a 
separate computer at the remote end in order to optimise transfers, so 
if you are just accessing a remote share using sshfs or similar then 
they can still work of course but as you realise they will be slow. I 
guess it is not possible for you to run rdiff-backup (or rsync) at the 
remote end as well?

You could run rdiff-backup locally to create a backup store and then 
mirror this store to the remote share using rcp. Still it will be slow 
because rdiff-backup always stores the latest copy of each file in full 
and so if this changes even slightly then the whole file will must be 
transferred by rcp.

Duplicity http://duplicity.nongnu.org/ might work better for you, 
because it uses forward diffs. Also its archives are secure.

Although not directly relevant I found a page here 
http://www.psc.edu/networking/projects/hpn-ssh/ which provides a patch 
to greatly speed up OpenSSH in some situations.