Re: [lmi] Sharing git repositories

[Vadim Zeitlin]

On Mon, 9 Mar 2020 23:21:51 +0000 Greg Chicares <address@hidden> wrote:

GC> We have several use cases, which are the cartesian product of:
GC> 
GC> (A) our single-user machines
GC> (B) a shared corporate server
GC> 
GC> crossed with
GC> 
GC> (0) The public lmi repository.
GC> (1) Cached repositories of wx and wxpdfdoc.
GC> (2) A proprietary repository of product parameters, containing
GC>  - *.policy, *.database, etc. files
GC>  - C++ code to generate those files
GC>  - corresponding *.rates files representing mortality tables
GC>  - system-test data: testdecks and results for proprietary products

[keeping the above for reference purposes]

GC> Even case B1 is "interesting", as these aren't bare repositories:
GC> 
GC> /srv/cache_for_lmi/vcs/wxWidgets[0]$git rev-parse --is-bare-repository
GC> false
GC> /srv/cache_for_lmi/vcs/wxpdfdoc[0]$git rev-parse --is-bare-repository
GC> false
GC> 
GC> Those are the result of deleting any cached copies and then running
GC> the normal scripts, e.g., in 'install_wx.sh':
GC> 
GC>   git clone "$coefficiency" --recurse-submodules "$remote_host_url" 
$repo_name
GC> 
GC> Should that be a bare repository instead?

 Well, "should" is a difficult question. What I can say is that right now,
it can't, because we use files from its working copy (first config.status,
then configure and then all the sources when building).

 However it does bother me that this repository, used for building lmi,
lives in a central location. This is not really compatible with the idea of
building lmi from different user accounts, and even though you found a
workaround for this with making the files group-accessible, it would still
be cleaner, IMHO, to just use different repositories for the different
users.

 In fact, if I were doing this afresh, I would have a pristine copy of the
upstream repository in some central location on the server and it would
indeed be bare. Then I would clone this repository to a per-used lmi build
directory and proceed from there.

 But this wouldn't be really compatible with the idea of having a single
script doing it all, as you'd need to set up the per-machine clone first
and then run the rest of install_wx.sh for each user. I also think that in
the future we really should be using Git submodules for wxWidgets anyhow,
so I'm not sure if we really want to be changing anything here.

GC> That's a bigger question. Ideally, Kim would be able to get rid of the
GC> cygwin installation on her msw laptop, and use this redhat server as a
GC> complete replacement:
GC> 
GC>  - server is much faster than company laptops
GC>  - no cygwin maintenance (issues with firewall and msw-10 compatibility)
GC>  - no version conflicts between cygwin and debian: we'd both use the
GC>    same exact version of debian (in a chroot)
GC> 
GC> And ideally I'd be able to use it too, although I'd still do most of
GC> my work on my own computer because that's much more convenient.

 All this seems eminently doable.

GC> Each of us potentially needs a local worktree. Of these ideas:
GC>  - separate, personal chroots, intended for single users only
GC>    (and maybe another for lmi's 'master' branch)
GC>  - separate directories in a single multi-user chroot
GC> which seems better?

 Using different users inside the same chroot seems like a natural approach
and I don't really see why would you want to do something much heavier with
per-user chroots, so I'd choose the latter.

GC> Given that it's a dedicated server, we might as well set up 'cron' jobs
GC> to keep a 'master' build current--e.g.,
GC>  - 'apt-get dist-upgrade' once a week
GC>  - build lmi daily

 FWIW I still have "Set up CI for lmi on GitHub" in my TODO list... And I'd
really, really like to do it, it's just that it's not a particular fun job
which will take a lot of time (because it will involve rebuilding
everything from scratch on relatively slow shared CI servers many times, as
I'm sure I will make many stupid mistakes).

GC> Then is the command we use to grab a wx repository (if not already cached):
GC>   git clone "$coefficiency" --recurse-submodules "$remote_host_url" 
$repo_name
GC> not the normal way? and if not, should we change it?

 This is the normal way because we work with this repository, and so we do
need to have a working directory. I don't think this contradicts what I
wrote but I realize now that we seem to be using different meanings of the
word "share": for me, "sharing a repository" means cloning from it, while
for you, I think, it means "working with the repository directly".

GC> > GC>   chgrp -R <group-name> .                   # Change files and 
directories' group
GC> > GC>   chmod -R g+w .                            # Change permissions
GC> > GC>   chmod g-w objects/pack/*                  # Git pack files should 
be immutable
GC> > GC>   find -type d -exec chmod g+s {} +         # New files get 
directory's group id
GC> [...]
GC> > GC> I think I understand the four chgrp and chmod commands above.
GC> > GC> But what does
GC> > GC>   git config core.sharedRepository group
GC> > GC> do that those four commands don't already do?
GC> > 
GC> >  If nothing else, it sets the sticky bit to ensure that all the new files
GC> > will be created with the correct (i.e. same) group too. It may have other
GC> > effects, but I'm not aware of them to be honest.
GC> 
GC> Exactly what do you mean by "sticky bit" here? In the context of the
GC> {rwxXst} bits for chmod, I think you mean setgid (s) rather than
GC> restricted-delete (t, the classic "sticky" bit).

 Oops, yes, I've mixed up my bits, sorry!

GC> But wouldn't this line:
GC> 
GC> > GC>   find -type d -exec chmod g+s {} +         # New files get 
directory's group id
GC> 
GC> above take care of the setgid (s) thing? Is it just that
GC>   git config core.sharedRepository group
GC> takes care of it in a clearer way?

 Yes, it does.

GC> Okay, it sounds like we should change this line in 'install_wx.sh':
GC> 
GC>   git clone "$coefficiency" --recurse-submodules "$remote_host_url" 
$repo_name
GC> 
GC> but what would we change it to?

 The change can't be limited to just a single line because we do need the
working copy. The simplest change would really be to just get rid of
/srv/cache_for_lmi/vcs completely and clone directly from the upstream
server to the local per-user directory. The main, and only, drawback of
this approach is that cloning the entire wx repository from GitHub or
Savannah will be slow. So the ideal solution, IMHO, would be to have a
single bare clone of this upstream repository somewhere on the server and
clone from it instead.

GC> That ideal presumes that a central repository exists. But none exists
GC> for our "proprietary" product repository.

 This is not a problem, of course, and you don't need to change anything
for this repository, but you could also decide to have a central copy of it
on this corporate server -- which looks like an ideal place for it.

 Of course, Git is decentralized and the fact that it's "central" is just a
societal convention, nothing more, so the day when you want to change it,
you can do it by just changing your workflow.


GC> For wx and wxpdfdoc, central repositories do exist, and we maintain
GC> a cached version of each locally (outside any chroot) from which we
GC> clone it into a chroot.

 Oops, I've somehow managed to completely miss the existence of these
repositories... It finally looks like you're already in my "ideal solution"
case from above.

GC> Does this sentence:
GC> 
GC> > [...] Your personal repositories don't need to set
GC> > core.sharedRepository at all.
GC> 
GC> apply to those cached versions?

 The pristine clone of the upstream repository should be bare and have
core.sharedRepository set for it. The clones of it created by install_wx.sh
should not be bare and should not have core.sharedRepository set.


 To finish this already long email, I'd just like to formulate what I think
are some simple general rules:

1. A repository should be bare if you want to only clone from and maybe
   push to, but definitely _not_ to modify the files, and hence directly
   create commits in, this repository.

2. A bare repository should be bare if you want to allow multiple users
   (who will typically all be members of the same Unix group) to clone from
   and, especially, push to this repository ("especially" because you can
   deal with cloning by just making the repository accessible to anyone,
   but you rarely want anyone to be able to modify the repository).

 I hope this helps to decide which repositories should, and should not, be
bare and/or shared, but please let me know if I still didn't manage to
express myself clearly.

 Good luck once again,
VZ

pgp0XVRY4Wxeg.pgp
Description: PGP signature