Re: [lmi] VCS caching

[Greg Chicares]

On 2018-04-16 23:58, Vadim Zeitlin wrote:
[...]
>  Sorry, but I'm afraid there are still a couple of problems left and I've
> created https://github.com/vadz/lmi/pull/82 to address them.
> 
>  The first two commits should be really uncontroversial as they just fix
> small but real mistakes when using git-clone and git-fetch respectively.

Cherry-picked into lmi master, and pushed.

> The last one is slightly more arguable as it only helps in my use case and
> not in yours (although it shouldn't hurt it neither, of course): without
> it, when I create the proxy wx repository in my VM, I still clone the
> submodules from GitHub and not from my local mirror. I'm really sorry for
> promising to get rid of the loop over submodules, I completely forgot that
> I still needed it for this. If you don't like it, you can skip applying
> this commit and I'll just tweak things locally.

Because of the way git has implemented submodules, it is easy to clone
a super-repository from a central server, but tricky to set up a proxy
for that central server. At least as git stands today, submodules embed
a central server into what would otherwise be a DVCS. In this full wx
clone, each submodule has a central server "registered":

git clone --jobs=32 --recurse-submodules 
https://github.com/wxWidgets/wxWidgets.git wxWidgets
Cloning into 'wxWidgets'...
Submodule '3rdparty/catch' (https://github.com/wxWidgets/Catch.git) registered 
for path '3rdparty/catch'
Submodule 'src/expat' (https://github.com/wxWidgets/libexpat.git) registered 
for path 'src/expat'
Submodule 'src/jpeg' (https://github.com/wxWidgets/libjpeg-turbo.git) 
registered for path 'src/jpeg'
Submodule 'src/png' (https://github.com/wxWidgets/libpng.git) registered for 
path 'src/png'
Submodule 'src/tiff' (https://github.com/wxWidgets/libtiff.git) registered for 
path 'src/tiff'
Submodule 'src/zlib' (https://github.com/wxWidgets/zlib.git) registered for 
path 'src/zlib'
Cloning into '/cache_for_lmi/vcs/wxWidgets/3rdparty/catch'...

and that registration is governed by versioned file '.gitmodules'. Of
course, a different server could be registered tomorrow on the central
server for any particular submodule, so this non-DVCS model is flexible.
But because a central server is hardcoded in a required versioned file,
git with submodules is not a distributed VCS. We want it to be--we try
to treat it as though it were--but it's not. The low-level API provides
enough flexibility to make it emulate a DVCS, but what the top-level
API lacks is a way to switch the registration dynamically--to redirect
the github references to a local mirror, e.g.:

git clone --redirect-server=[editor-subcommand]
Submodule 'src/zlib' registration...processing '--redirect-server' option...
    subcommand: 's!https://github.com/wxWidgets/!/my/local/mirror/zlib.git!'
    originally: https://github.com/wxWidgets/zlib.git
    redirected: /my/local/mirror/zlib.git
Cloning into '/cache_for_lmi/vcs/wxWidgets/src/zlib' from 
'/my/local/mirror/zlib.git'...

Such a mirror might be a directory, a filesystem mount, or an SSH or
HTTPS URL. We developed a script to do that, through the lower-level
git-submodule API, for multiple use cases. It took a lot of time and
effort to debug and test, but in the end we had a script that was
proven to work.

Then we rewrote it for simplicity. That came at a cost: without our
own explicit loop, we can't change the embedded github URL to...
 - reference our own local mirror's submodules
 - reference our own local bare repositories for submodules
For example, I had set up a 400 MB collection of mirrored bare
repositories, and I traded that for one 630MB non-bare repository
in order to be able to use the top-level git-clone API to recurse
through submodules automatically. The tradeoff was:
 - complicated commands, minimal data, maximum flexibility; vs.
 - simple commands, cluttered data, low flexibility.

The reason why I don't want to apply this particular change is
that it seems to restore only some of the careful, validated work
we had done. If we go back to complexity, I want to recapture
everything we had bought in return. Otherwise, I want to keep the
simplicity we traded for. Looking at this crucial diff:
  git diff 1dceecb0 e4c83688
I can't just revert it, because, while it was carefully tested,
it had turned to spaghetti and needed a rewrite. If I revert it,
I have to rewrite it again, and then begin another testing cycle.
Sorry, I just can't make time for that.

Let me offer a suggestion that I think will reduce your use case
to the one that the simplified 'install_wx.sh' fully supports:
write a separate script to clone the repository you keep on your
LAN into your /cache_for_lmi/vcs directory. Do that once, and
then that directory is a complete proxy; it shouldn't need to
connect to the central server on github unless submodules change.