Re: [rdiff-backup-users] rdiff-backup-users Digest, Vol 153, Issue 2

[Carter J. Castor]

unsubscribe rdiff-backup-users
Carter J. Castor


On Sat, Aug 15, 2015 at 3:01 AM,  <address@hidden> wrote:
> Send rdiff-backup-users mailing list submissions to
>         address@hidden
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
> or, via email, send a message with subject or body 'help' to
>         address@hidden
>
> You can reach the person managing the list at
>         address@hidden
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of rdiff-backup-users digest..."
>
>
> Today's Topics:
>
>    1. Re: State of the rdiff-backup project (Robert Nichols)
>    2. Re: State of the rdiff-backup project (Tobias Leupold)
>    3. Re: State of the rdiff-backup project (Claus-Justus Heine)
>    4. Re: State of the rdiff-backup project (Tobias Leupold)
>    5. Re: State of the rdiff-backup project (Dominic Raferd)
>    6. Re: State of the rdiff-backup project (Claus-Justus Heine)
>    7. Re: State of the rdiff-backup project (Claus-Justus Heine)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 14 Aug 2015 12:12:18 -0500
> From: Robert Nichols <address@hidden>
> To: address@hidden
> Subject: Re: [rdiff-backup-users] State of the rdiff-backup project
> Message-ID: <address@hidden>
> Content-Type: text/plain; charset=windows-1252; format=flowed
>
> On 08/13/2015 02:16 PM, Claus-Justus Heine wrote:
>>  I'm really no python guy. But: I really
>> would like to have the "regression takes ages" and "crowded directories
>> take ages" being resolved. OR OR OR: would like to understand this
>> issue. Cannot claim that I will be able to fix it. But I would very much
>> like to understand what the heck is going on there. Just doesn't feel sa
>> ne.
>
> I doubt that the "regression takes ages" problem can be fixed within
> rdiff-backup. It's inherently a complex operation that requires
> searching throughout the archive for things that aren't consistent with
> the previous state. Remember that you can't trust that the latest
> metadata files are consistent with the current state of the mirror and
> increments.  In large part it's due to use of the filesystem as a
> database, with bits of information scattered in file names in the
> increments directory and various metadata files. You're not going to
> change that without a major rewrite.
>
> I suppose one solution to the regression issue is to store the archive
> in a filesystem or LVM volume that supports snapshots.  Rather than let
> rdiff-backup do the regression, stop it and restore the snapshot. I
> suspect the penalty in space (transient, until the snapshot is deleted)
> and performance for the backup would be serious. And that still leaves
> the issue of regressing more than the last level.
>
> Like you, I'm no Python guy. Every time I try to study it, I end up as a
> lump in a snake's belly. I think it's because there are some things
> about the language that I hate (starting with the use of whitespace as a
> syntax element) and the incompatibility of major versions. And then
> there is the tendency of Python programmers to believe that stack
> backtraces are an acceptable substitute for meaningful error messages.
> It all leaves a bad taste that I just can't get around.
>
> --
> Bob Nichols     "NOSPAM" is really part of my email address.
>                  Do NOT delete it.
>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Fri, 14 Aug 2015 19:30:49 +0200
> From: Tobias Leupold <address@hidden>
> To: address@hidden
> Subject: Re: [rdiff-backup-users] State of the rdiff-backup project
> Message-ID: <address@hidden>
> Content-Type: text/plain; charset="us-ascii"
>
>> You're not going to change that without a major rewrite.
>
>> Like you, I'm no Python guy.
>
> I like Python. And I like rdiff-backup. But perhaps, some skilled programmer
> has the energy to take the "chance" of the questionable state of rdiff-backup
> by only continuing the concept of rdiff-backup and rewriting some similar
> program from scratch, e. g. in C++. Perhaps with some metadata database to
> solve the regression issue (which really sucks). And so on.
>
> Not that I would be remotely skilled enough in C++ to do so ... I'm only
> thinking about what could be done ...
>
>
>
> ------------------------------
>
> Message: 3
> Date: Fri, 14 Aug 2015 20:43:18 +0200
> From: Claus-Justus Heine <address@hidden>
> To: address@hidden
> Subject: Re: [rdiff-backup-users] State of the rdiff-backup project
> Message-ID: <address@hidden>
> Content-Type: text/plain; charset=utf-8
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Am 14.08.2015 um 19:30 schrieb Tobias Leupold:
>>> You're not going to change that without a major rewrite.
>>
>>> Like you, I'm no Python guy.
>
> I do not have any serious objections against Python. Its widely used,
> and the limiting time factor is IO efficiency, and not the question
> about the fasted for-loop ever possible.
>
> Concerning a "major rewrite": for me this is out of scope. I was rather
> thinking about a code review. I first want to figure out whether there
> are maybe places which can be optimized significantly, e.g. excessive
> readir() stuff or something like this, or any other duplicate file IO
> which doesn't hurt on small backup sets. Also before a major code
> rewrite could be tackled at all, one would first have to understand what
> rdiff-backup is doing beneath its skin. Otherwise it would not be clear
> what the precise goal of such a major rewrite should be. Ok: to produce
> something not worse than rdiff-backup. But to clarify this, you first
> have to understand the existing program.
>
> Cheers,
>
> Claus
>
>>
>> I like Python. And I like rdiff-backup. But perhaps, some skilled prog
> rammer
>> has the energy to take the "chance" of the questionable state of rdiff
> - -backup
>> by only continuing the concept of rdiff-backup and rewriting some simi
> lar
>> program from scratch, e. g. in C++. Perhaps with some metadata databas
> e to
>> solve the regression issue (which really sucks). And so on.
>>
>> Not that I would be remotely skilled enough in C++ to do so ... I'm on
> ly
>> thinking about what could be done ...
>>
>> _______________________________________________
>> rdiff-backup-users mailing list at address@hidden
>> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
>> Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBac
> kupWiki
>>
>
>
> - --
> Claus-Justus Heine                      address@hidden
>
> http://www.claus-justus-heine.de/
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
>
> iQIcBAEBCAAGBQJVzjbEAAoJEF8rbXubOwvK7sAP/iyuj+1CGjOBt1VMUAf1Tr88
> aM3/a1VevcwjLQ/KjyrExMYlzYI5tOU9Dx+JcB2pJzvEIcvb9vBl32hO0CUNUHvt
> k+yz6D8C7nkXwZoZWDZH0Ct3yMPYfMCX/pa5B1OqrPLHaRrG6QIWNubDRlf8pKJL
> e+lnQyyhfdOY1OBLH2GGQAuH4dNsTAwZYkXrvw1n0QA1mXU6cqbacZdv6mFi4wCb
> O7gmzbA96agxmh9Z0zSg520IS9FPt0e2uwMzgTTKdOUX4zqD09v9/wWTGJSGjhhp
> NQ3kQGylQRiPZAhnDzyeWjLy3/3nivj2F3aBcymhNZOqpVmtqgxEYgg9Ck6AokQM
> dHmimEG6GR9VtxpU/MfwSAPXhNrr5IktofdyLebc+GYDmq/RlZmCoFl2WM70Prhw
> 6PU39kvC2Oa44whmJIK4aAyVUr9q92T5hv3VT+dI5vRh0rBSfDW9r2ypnMMu1+n5
> GpfBU0ndOAOboqsufcyDBm5xakTthzFMS1mxFJq4dJjNdO/17naZNm3DG5O4d7hE
> CDoCZ6tb+o3OipxF7AOoX0ZoxMs+lGlEOOHZx0pn3YW1oiAod9E6LYm9ba84dxAK
> 7NJfgqE9NGzuh8AP2GV099l8Ces32qNHI17W2n3IfMVwlt7VMXGvKidlR+vsVcwv
> d1QGXv7KFo9Q+0y0+kLz
> =qJ/a
> -----END PGP SIGNATURE-----
>
>
>
> ------------------------------
>
> Message: 4
> Date: Fri, 14 Aug 2015 20:50:56 +0200
> From: Tobias Leupold <address@hidden>
> To: address@hidden
> Subject: Re: [rdiff-backup-users] State of the rdiff-backup project
> Message-ID: <address@hidden>
> Content-Type: text/plain; charset="us-ascii"
>
>> But to clarify this, you first  have to understand the existing program.
>
> Of course! And speaking of me, I don't ... I was just thinking about the whole
> situation.
>
>
>
> ------------------------------
>
> Message: 5
> Date: Fri, 14 Aug 2015 19:53:42 +0100
> From: Dominic Raferd <address@hidden>
> To: address@hidden
> Subject: Re: [rdiff-backup-users] State of the rdiff-backup project
> Message-ID: <address@hidden>
> Content-Type: text/plain; charset=utf-8; format=flowed
>
> My 2p...
>
> Rdiff-backup has limitations and it would be *really* good if someone
> who understood python could step up and maintain the code, but I don't
> see the problem with regressions. Yes they are slow but they should be
> emergency operations reserved for rare circumstances. If you are
> frequently experiencing broken backup session then I think you should
> look at why that is happening. We only use rdiff-backup within our lan
> and backups always complete. I have only needed regressions when we have
> inadventently backed up a lot of extraneous data, when I use them to
> overcome bloating of the repository. For offsite backup (of the entire
> set of repositories) we use rsync. I don't find backup speeds with
> rdiff-backup particularly slow BTW, but we run the backups at a quiet
> time when speed is not critical.
>
> v1.2.8 is the official stable version, that's why Debian uses it. v1.3.3
> is officially 'unstable' although I haven't heard of any problems with
> it (and haven't used it).
>
> I keep repositories on ext4 - I tried btrfs but found it very slow (on
> our vm) and the big wins that I sought (compression, deduplication) made
> it slower still - and btrfs deduplication is still buggy.
>
> Dominic
> http://www.timedicer.co.uk
>
>
>
> ------------------------------
>
> Message: 6
> Date: Fri, 14 Aug 2015 20:55:45 +0200
> From: Claus-Justus Heine <address@hidden>
> To: address@hidden
> Subject: Re: [rdiff-backup-users] State of the rdiff-backup project
> Message-ID: <address@hidden>
> Content-Type: text/plain; charset=utf-8
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Am 14.08.2015 um 20:50 schrieb Tobias Leupold:
>>> But to clarify this, you first  have to understand the existing progr
> am.
>>
>> Of course! And speaking of me, I don't ... I was just thinking about t
> he whole
>> situation.
>
> I tried to give it a round a couple of years ago, and gave up on it for
> lack of time. And I do not know how things will be going on this time.
> We will see.
>
> Cheers,
>
> Claus
>
>>
>> _______________________________________________
>> rdiff-backup-users mailing list at address@hidden
>> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
>> Wiki URL: http://rdiff-backup.solutionsfirst.com.au/index.php/RdiffBac
> kupWiki
>>
>
>
> - --
> Claus-Justus Heine                      address@hidden
>
> http://www.claus-justus-heine.de/
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
>
> iQIcBAEBCAAGBQJVzjmuAAoJEF8rbXubOwvKhoEP/21Or6fBcSaSesR8jJn/l+A0
> F1wUOu74ogcy2WzXDP1sBv2cSu3gevVeYBbQJOGrOafeBk7oXbkagGH/u6JKjCGy
> 6EhMzjjwLaehniU0hdlQB7XsjxMFE7Js8bJlnSMDgHHWvMpgsadOtPrl+qJ189nY
> b181rTSdRmHSpFYxjCgjf5jcckwS7M9XkLpmz7/BJRzXh6pp43iIdJH8STd/4+1b
> w80ApGs7eYbbi+Fh87y/dNNnVN4LpvZjZSB6qpPQR5f6TQ93EHoJRu3t4cHphzqZ
> RN5iwK3jOygb6noCUiu3D0dRtCGccgP5JXZNrBLkyBDBzkdWcF/6Bd1XKwq1PmEH
> 9bjZld2HgdIA/3eUQUodJ1X2Hl27EZfSy8xrARbdLhJkK5nf1L7IemkUfYkCWTq3
> Dl7sQ5X56KlyAgIDTxcPoh4vx7U7/IUXI1BILF/f0/vU3wgNnrImihBns380Udye
> sm1TZsqKiQ94HViU1rW5VMx3ITk3dzAZiZ+Osml+r0LGzLMslCR8YFDTw6vZsA25
> 3oh/9pp29ZXNoRAgVWyCc8roCg/P2ko72DYf78qenGrWoU0YpMsxmA6mUKIcKw6k
> La/1XUJZBqSsH2wmX/0BPrbsgIz5QWlob6bGafzDs2l4LoP9x9fsv4OtP5emrg7Q
> X5L1lxju197veLHUQuyM
> =n9Sq
> -----END PGP SIGNATURE-----
>
>
>
> ------------------------------
>
> Message: 7
> Date: Fri, 14 Aug 2015 21:00:54 +0200
> From: Claus-Justus Heine <address@hidden>
> To: address@hidden
> Subject: Re: [rdiff-backup-users] State of the rdiff-backup project
> Message-ID: <address@hidden>
> Content-Type: text/plain; charset=utf-8
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> Am 14.08.2015 um 20:53 schrieb Dominic Raferd:
>> My 2p...
>>
>> Rdiff-backup has limitations and it would be *really* good if someone
>> who understood python could step up and maintain the code, but I don't
>> see the problem with regressions. Yes they are slow but they should be
>> emergency operations reserved for rare circumstances. If you are
>> frequently experiencing broken backup session then I think you should
>> look at why that is happening. We only use rdiff-backup within our lan
>> and backups always complete. I have only needed regressions when we ha
> ve
>> inadventently backed up a lot of extraneous data, when I use them to
>> overcome bloating of the repository. For offsite backup (of the entire
>> set of repositories) we use rsync. I don't find backup speeds with
>> rdiff-backup particularly slow BTW, but we run the backups at a quiet
>> time when speed is not critical.
>
> Well, I'am actually in progress of figuring things out on my side.
> However, I experience slowness at times. Also: even if regressions are
> extra-ordinary events, it still does not feel right that they take so
> long. At least I want to try to understand what is going on there.
>
>>
>> v1.2.8 is the official stable version, that's why Debian uses it. v1.3
> .3
>> is officially 'unstable' although I haven't heard of any problems with
>> it (and haven't used it).
>
> Silly me. You are right. Maybe I should volunteer as maintainer and as
> first act copy v1.3.3 tp v1.4.0 and declare it stable ;)
>
>> I keep repositories on ext4 - I tried btrfs but found it very slow (on
>> our vm) and the big wins that I sought (compression, deduplication) ma
> de
>> it slower still - and btrfs deduplication is still buggy.
>
> For my "slowness" experience this may very well be the problem, at the
> moment I'm running btrfs. However, as I'm right now replacing the backup
> hardware this would be the time to change something. We will see.
>
> Many thanks for your feed-back,
>
> Claus
>
>
> - --
> Claus-Justus Heine                      address@hidden
>
> http://www.claus-justus-heine.de/
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
>
> iQIcBAEBCAAGBQJVzjrkAAoJEF8rbXubOwvKlZIQALXxUAme+GPIMxxMGUThpMsg
> P2EjSTCZeJXRWMHg3obGLWFg7eVf3L2NNuPd/i94rEyHTVQFNpXYC5+aTH3I9eKM
> 8GnapWqlZWF2ldfw4GLe8RGzm45e+WkeVF3MkiwIaMTQJ1jtOIF/kEtmeNgMWpeo
> Mt8H/J3Rzk3Iz+zRbI+nUKRLq0ZQ9LDpiL+ob+IdnLSHSKT6LKiQz4o6VKHfimf6
> EcfuaecD6J6HRoaPOTcepk73r3IgjFHMx5yYdGveav7TbloBiNJqycaxS3XXfNLf
> I3QBJt0AqXJ4dnXSv77XrpnyysMCpDcLFQEwncU5q7iOBEB9gpeOWt5TGpteawXP
> 4W+Mw5hqnOK0f37EwwdKEVU6KUU87l/jOkMpO3uPYEjdX9ukykeLqwGq3gZakSry
> 0tGrzOvUuzXouIFW0bFNyBjQv1WBGHupzEm/R3sShoLe1Yn7G2viUt8dgLioNvAq
> IvoD80q18bRElzWHfY1iHgno3lIkqIw4XKeEMw7SLydW74warYjBimVxnJ7Mf7jp
> uafhW1hrqUVkKspWEmfLJ64wrEEJsecfTV7MGaJsybZ8zTg2OhS39H08gqPXQcc3
> DxWmzyK+ZMSx3fV3puXT7IBQdBq280Sd/DxnLDmf0P05PM8A2zPheuvr/h/CpZma
> 88qGbv58T6MN2rNkKwKA
> =0HCf
> -----END PGP SIGNATURE-----
>
>
>
> ------------------------------
>
> _______________________________________________
> rdiff-backup-users mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/rdiff-backup-users
>
>
> End of rdiff-backup-users Digest, Vol 153, Issue 2
> **************************************************