Re: [lmi] VCS caching

[Vadim Zeitlin]

On Fri, 13 Apr 2018 22:51:29 +0000 Greg Chicares <address@hidden> wrote:

GC> I imagine you might run the WX script by overriding $wx_git_url, e.g.:
GC>   $ wx_git_url='/my/lan/directory' install_wx.sh
GC> But I'd like you to be able to run the MSW script without unpleasant
GC> surprises like having it download 300 MiB from github that you had
GC> already mirrored locally. Right now the MSW script includes the line:
GC>   ./install_wx.sh
GC> and if you run the MSW script right now, I think you will get that
GC> gigantic download because of the way it invokes the WX script.

 Yes, indeed.

GC> Please review commit c253ef10241, which I'll inline below to make it
GC> easier to comment. I think that if you use something like a mount to
GC> make the above usage
GC>   $ wx_git_url='/my/lan/directory' install_wx.sh
GC> translate to
GC>   $ wx_git_url='/cache_for_lmi/vcs' install_wx.sh
GC> then, with this change, the MSW script should DTRT for you.

 I don't use a mount, my repository is accessed via ssh, as usual, but it
still should work AFAICS.

GC> I'm especially interested in your thoughts on new submodules that
GC> might be added later--see the comments embedded in the change below.
GC> Already I'm having second thoughts: it's trivial to copy a submodule's
GC> .git directory to the cache directory, but it's not so trivial to find
GC> that .git directory if you don't already know where it is.

 But you can always find where it is by using "git config", either for this
particular submodule or for all of them at once, e.g.:

        $ git config -f .gitmodules --get-regexp 'submodule\..*\.path'


GC> 
---------8<--------8<--------8<--------8<--------8<--------8<--------8<-------
GC> commit c253ef10241c0be835429023535bb44067aec3d3 (HEAD -> master)
GC> Author: Gregory W. Chicares <address@hidden>
GC> Date:   2018-04-13T22:39:46+00:00
GC> 
GC>     Use cached local wx git repository if found
GC>     
GC>     This is a request for review. It has been tested, so it's okay for
GC>     production, but there may be better ways to accomplish the purpose.
GC> 
GC> diff --git a/install_wx.sh b/install_wx.sh
GC> index 8922ac30..fe518ef6 100755
GC> --- a/install_wx.sh
GC> +++ b/install_wx.sh
GC> @@ -23,9 +23,36 @@
GC>  
GC>  set -vxe
GC>  
GC> +# Preliminarily check local cache 
##############################################
GC> +
GC> +# Repositories are kept in /cache_for_lmi/vcs/ whence they can be
GC> +# cloned cheaply and reliably--whereas cloning them from a remote
GC> +# host takes considerable time and bandwidth, and fails if internet
GC> +# connectivity is lost, or the host is temporarily unavailable, or
GC> +# it is blocked by a corporate firewall.
GC> +
GC> +wx_repository="wxWidgets.git"
GC> +default_server="https://github.com/wxWidgets";
GC> +default_url="$default_server"/"$wx_repository"
GC> +
GC> +cache_dir="/cache_for_lmi/vcs"
GC> +mkdir --parents "$cache_dir"
GC> +
GC> +if git ls-remote "$cache_dir"/"$wx_repository" >/dev/null

 I'm sorry, I must be missing something obvious here, but what exactly does
this command do and why do we need it? Is the intention here to just check
whether $cache_dir/$wx_repository exists? If so, there is a dedicated
command to do just this: "git rev-parse --is-bare-repository" (you could
omit the "--is-bare-repository" part if you wanted to just check whether
the current directory is any kind of Git repository).


On Sat, 14 Apr 2018 00:09:06 +0000 Greg Chicares <address@hidden> wrote:

GC> On 2018-04-13 22:51, Greg Chicares wrote:
GC> > [...]
GC> > +    default_url="$cache_dir"/"$wx_repository"
GC> > +    # Now we could make a list of submodules...
GC> > +    cd "$default_url"
GC> > +    git show master:.gitmodules |sed -e'/\.git$/!d' -e's!.*\/!!'
GC> > +    # ...and download any that are missing to local cache, but
GC> > +    # that seems awfully fancy. Instead, we'll let the code that
GC> > +    # follows download them to our clone, and if we're mindful,
GC> > +    # we'll move them from there to the cache.
GC> 
GC> commit 330ca8f6f714 automates that.

 Note that by doing it like this, you lose the benefit of updating the
submodules in parallel added in ad8935088005739eaefcc810c3dc46fff0261cb8
So I think it would be better, as it would make the initial checkout
faster, to keep doing "git submodule update --jobs $(nproc) --init", and
then "git clone --bare" the submodules missing from the cache directory
there.

 Regards,
VZ