Re: [lmi] VCS caching

[Vadim Zeitlin]

On Sat, 14 Apr 2018 16:39:19 +0000 Greg Chicares <address@hidden> wrote:

GC> On 2018-04-14 01:05, Vadim Zeitlin wrote:
GC> > On Fri, 13 Apr 2018 22:51:29 +0000 Greg Chicares <address@hidden> wrote:
GC> [...]
GC> > GC> Please review commit c253ef10241
GC> [...]
GC> >  I don't use a mount, my repository is accessed via ssh, as usual, but it
GC> > still should work AFAICS.
GC> 
GC> That affects the next paragraph...
GC> 
GC> > GC> +if git ls-remote "$cache_dir"/"$wx_repository" >/dev/null
GC> > 
GC> >  I'm sorry, I must be missing something obvious here, but what exactly 
does
GC> > this command do and why do we need it? Is the intention here to just check
GC> > whether $cache_dir/$wx_repository exists?
GC> 
GC> Yes, exactly.
GC> 
GC> > If so, there is a dedicated
GC> > command to do just this: "git rev-parse --is-bare-repository" (you could
GC> > omit the "--is-bare-repository" part if you wanted to just check whether
GC> > the current directory is any kind of Git repository).
GC> 
GC> git-ls-remote works in all these cases:
GC> 
GC> /srv/cache_for_lmi/vcs[0]$git ls-remote 
/srv/cache_for_lmi/vcs/wxWidgets.git >/dev/null
GC> /srv/cache_for_lmi/vcs[0]$git ls-remote 
https://github.com/wxWidgets/wxWidgets.git >/dev/null
GC> /srv/cache_for_lmi/vcs[0]$git ls-remote 
ssh://github.com/wxWidgets/wxWidgets.git >/dev/null 
GC> Permission denied (publickey).

 I think I see the problem now: you want to use the cache directory
unconditionally while I was thinking of it as a traditional, i.e.
optional, cache. So for me we only need to test whether a (local) cache
directory exists and for this git-rev-parse is perfectly sufficient.

 Just to be clear, personally I don't want to use any cache directory
inside my lmi VM at all, this is just an unnecessary complication because
for me cloning from the existing LAN repository is quick enough anyhow.
This being said, if it's absolutely indispensable to use a cache directory
for all the other lmi developers, I certainly would find a way to live with
it, but in this case I would strongly suggest to: (a) not call this
directory a cache directory, if it's not optional and (b) explain why
git-ls-remote is used here because it's really not obvious.

GC> > On Sat, 14 Apr 2018 00:09:06 +0000 Greg Chicares <address@hidden> wrote:
GC> >   [...we ought to automatically clone any missing submodules...]
GC> > GC> commit 330ca8f6f714 automates that.
GC> > 
GC> >  Note that by doing it like this, you lose the benefit of updating the
GC> > submodules in parallel added in ad8935088005739eaefcc810c3dc46fff0261cb8
GC> > So I think it would be better, as it would make the initial checkout
GC> > faster, to keep doing "git submodule update --jobs $(nproc) --init", and
GC> > then "git clone --bare" the submodules missing from the cache directory
GC> > there.
GC> 
GC> I'm not sure I follow. My intention was:
GC> 
GC>   for(auto& i : submodules(".gitmodules"))
GC>     {
GC>     cache_url = i, with https://github... replaced by /cache_for_lmi/...
GC>     if $cache_url does not exist
GC>       clone i into $cache_url
GC>     plug $cache_url into .gitmodules
GC>     }

 I realize it now, but this is very different from the initial logic and
also rather different from my understanding about how a cache works, so it
took me some time to understand it, sorry for the confusion.

 Nevertheless my point above is still true because the "clone i" steps
above are executed sequentially, unlike "git submodule update --jobs" which
would perform them in parallel. So the first execution of the script would
still be (possibly significantly) slower.

GC>   // Now the invariants for parallel 'git submodule update' are true:

 Yes, but it's too late. The part which really takes time and is worth
being optimized is cloning from the remote server.


 I guess the real problem is that I'm still not sure about what exactly are
we trying to accomplish. For a simple deployment of lmi, on a single MSW
machine, having this mandatory cache doesn't seem to be useful at all
(please correct me if I'm wrong). For your deployment of lmi, inside one of
possibly multiple chroots on a Linux system, it doesn't seem to have any
advantages compared to just creating the real cache once on the host system
and then either mounting it, with bind mount option, into each chroot and
cloning from it or using ssh://localhost to access the cache. In my case,
it's not useful neither.

 So I still don't really see why are we doing all this. The initial version
of the script was rather simple and its logic was pretty straightforward:
clone wxWidgets and all its submodules from the given URL, which could
point either to the upstream repository location or some local mirror. The
current version is somewhat less simple and its logic is more opaque and
for no gain I can identify. Why do we absolutely need to have this local
cache that we must be able to modify instead of just using it if it's
available?

 I'm sorry if it's another question the answer to which is absolutely
obvious for you, but, in my defense, it's really not very clear from the
outside point of view.

 Regards,
VZ