Re: [lmi] VCS caching

[Greg Chicares]

On 2018-04-14 01:05, Vadim Zeitlin wrote:
> On Fri, 13 Apr 2018 22:51:29 +0000 Greg Chicares <address@hidden> wrote:
[...]
> GC> Please review commit c253ef10241
[...]
>  I don't use a mount, my repository is accessed via ssh, as usual, but it
> still should work AFAICS.

That affects the next paragraph...

> GC> +if git ls-remote "$cache_dir"/"$wx_repository" >/dev/null
> 
>  I'm sorry, I must be missing something obvious here, but what exactly does
> this command do and why do we need it? Is the intention here to just check
> whether $cache_dir/$wx_repository exists?

Yes, exactly.

> If so, there is a dedicated
> command to do just this: "git rev-parse --is-bare-repository" (you could
> omit the "--is-bare-repository" part if you wanted to just check whether
> the current directory is any kind of Git repository).

git-ls-remote works in all these cases:

/srv/cache_for_lmi/vcs[0]$git ls-remote /srv/cache_for_lmi/vcs/wxWidgets.git 
>/dev/null
/srv/cache_for_lmi/vcs[0]$git ls-remote 
https://github.com/wxWidgets/wxWidgets.git >/dev/null
/srv/cache_for_lmi/vcs[0]$git ls-remote 
ssh://github.com/wxWidgets/wxWidgets.git >/dev/null 
Permission denied (publickey).

(except the last, because of an issue that shouldn't affect SSHing into
your own machine).

I don't see how to make git-rev-parse work in all those cases:

/srv/cache_for_lmi/vcs[128]$git rev-parse /srv/cache_for_lmi/vcs/wxWidgets.git 
fatal: Not a git repository (or any of the parent directories): .git
/srv/cache_for_lmi/vcs[128]$git rev-parse 
https://github.com/wxWidgets/wxWidgets.git
fatal: Not a git repository (or any of the parent directories): .git
/srv/cache_for_lmi/vcs[128]$git rev-parse 
ssh://github.com/wxWidgets/wxWidgets.git
fatal: Not a git repository (or any of the parent directories): .git

I can make it work with a mounted filesystem:

/srv/cache_for_lmi/vcs[0]$git -C zlib.git rev-parse                       
/srv/cache_for_lmi/vcs[0]$git -C /srv/cache_for_lmi/vcs/zlib.git rev-parse

but I don't know how to make 'git -C' work with a remote server:

/srv/cache_for_lmi/vcs[128]$git -C wxWidgets.git rev-parse 
https://github.com/wxWidgets           
https://github.com/wxWidgets
fatal: Invalid object name 'https'.

> On Sat, 14 Apr 2018 00:09:06 +0000 Greg Chicares <address@hidden> wrote:
>   [...we ought to automatically clone any missing submodules...]
> GC> commit 330ca8f6f714 automates that.
> 
>  Note that by doing it like this, you lose the benefit of updating the
> submodules in parallel added in ad8935088005739eaefcc810c3dc46fff0261cb8
> So I think it would be better, as it would make the initial checkout
> faster, to keep doing "git submodule update --jobs $(nproc) --init", and
> then "git clone --bare" the submodules missing from the cache directory
> there.

I'm not sure I follow. My intention was:

  for(auto& i : submodules(".gitmodules"))
    {
    cache_url = i, with https://github... replaced by /cache_for_lmi/...
    if $cache_url does not exist
      clone i into $cache_url
    plug $cache_url into .gitmodules
    }
  // Now the invariants for parallel 'git submodule update' are true:
  assert(every submodule is cached)
  assert(every url in .gitmodules has been replaced with a cached url)
  // ...so the update can be performed in parallel:
  git submodule update --recursive --jobs $(nproc) --init

The "if not already cached" step clones into the cache directory.
The recursive parallel update clones from the cache directory into
the local directory. For the demonstration below, I purge the cached
zlib clone, then run the script. First it clones the missing zlib
into cache...

fatal: Cannot change to '/cache_for_lmi/vcs/zlib.git': No such file or directory
+ git clone --bare https://github.com/wxWidgets/zlib.git 
/cache_for_lmi/vcs/zlib.git
Cloning into bare repository '/cache_for_lmi/vcs/zlib.git'...

...then it points .gitmodules to that newly cached clone...

+ git config submodule.src/zlib.url /cache_for_lmi/vcs/zlib.git

...and then 'git submodule update' clones all submodules from
cache into local in parallel, treat zlib the same as all others,
in particular updating from:
  '/cache_for_lmi/vcs/zlib.git'
into:
  '/tmp/vcs/wxWidgets/src/zlib'

Cloning into '/tmp/vcs/wxWidgets/src/tiff'...
done.
Cloning into '/tmp/vcs/wxWidgets/src/zlib'...
done.

In case that synopsis is too brief, here's a less-abbreviated
screen capture:

/tmp/vcs[0]$cd /tmp; rm -rf /tmp/vcs/wxWidgets; cd /tmp/vcs
/tmp/vcs[0]$ls /cache_for_lmi/vcs                          
Catch.git  libexpat.git  libjpeg-turbo.git  libpng.git  libtiff.git  
wxWidgets.git  zlib.git
/tmp/vcs[0]$rm -rf /cache_for_lmi/vcs/zlib.git             
/tmp/vcs[0]$/opt/lmi/src/lmi/install_wx.sh            

git submodule status | grep '^-' | cut -d' ' -f2 | while read -r subpath
do
    suburl=$(git config --file .gitmodules --get submodule.${subpath}.url)

    # If the submodule hasn't been cached yet, clone it to cache now.
    cache_url="$cache_dir"/${suburl##*/}
    if ! git -C "$cache_url" rev-parse
    then
        git clone --bare "$suburl" "$cache_url"
    fi
...
    git config submodule.${subpath}.url ${wx_git_url%/*}/${suburl##*/}
done
+ git submodule status
+ grep ^-
+ cut+  -d  -f2
read -r subpath
...
+ git config --file .gitmodules --get submodule.src/zlib.url
+ suburl=https://github.com/wxWidgets/zlib.git
+ cache_url=/cache_for_lmi/vcs/zlib.git
+ git -C /cache_for_lmi/vcs/zlib.git rev-parse
fatal: Cannot change to '/cache_for_lmi/vcs/zlib.git': No such file or directory
+ git clone --bare https://github.com/wxWidgets/zlib.git 
/cache_for_lmi/vcs/zlib.git
Cloning into bare repository '/cache_for_lmi/vcs/zlib.git'...
...
+ git config submodule.src/zlib.url /cache_for_lmi/vcs/zlib.git
+ read -r subpath
...
git submodule update --recursive --jobs $(nproc) --init
+ nproc
+ git submodule update --recursive --jobs 32 --init
...
Cloning into '/tmp/vcs/wxWidgets/src/tiff'...
done.
Cloning into '/tmp/vcs/wxWidgets/src/zlib'...
done.