[RFC] Enabling GitLab CI

[Jonas Hahnfeld]

I'd like to propose that we enable GitLab CI and use it for
automatically testing our merge requests. It would run 'make', 'make
test', and 'make doc' like the current manual testing process. Once we
have that, GitLab can be configured to enforce a successful pipeline
before merging. As we only allow fast-forward merges, this means each
MR has received testing in the form it hits master. This would
effectively replace the current setup of pushing to staging.

The system would use Docker containers which is similar in spirit to
Han-Wen's proposal in February [1]. However the system is much simpler
and the containers don't come with a full development environment.
Instead they have barely enough to compile and test LilyPond.
For the actual changes and to get a feel of the UX, see the MR:
https://gitlab.com/lilypond/lilypond/-/merge_requests/57


Advantages
==========

 - We get automatic testing, tightly integrated into GitLab.
 - It helps the Patch Meister (James); see below for one caveat.
 - No patchy required for pushing to master.


Disadvantages
=============

 - Using Docker images, we get a single standardized environment for
the tests. While being great for reproducibility, it looses the
diversity we had so far (please refer to earlier discussions).
 - Comparison of regression tests is not yet integrated, the main
problem being the need for a baseline. I already have an idea or two
how this could work, but for now I'm focusing on the initial setup.
This means James still needs to download the patches and run 'make
check' (make doc being run automatically).
 - There's no possibility to push without a merge request, and it needs
to be merged through the UI. I don't consider this a loss personally.
 - There's no possibility to push multiple changes at once: Merge
requests are rebased, tested and merged one by one. GitLab is planning
to implement a queuing mechanism, and if all else fails we can invent
our own scripts.


Where to Run
============

Now towards infrastructure: Being a public project and open-source
GitLab currently offers 50,000 CI minutes per month for free. [1] This
has the advantage that test results are available immediately.
50,000 minutes divided by ~60 minutes per build, this is ~800 builds
per month, or around 25 per day. Not sure what happens if we exceed
this, comments claim it's "enforced" in some way. However there's no
page to see the current usage (for a public group project)...

If this is too little, if GitLab reduces the amount of free minutes, or
if we want to get faster builds, we can always add our own machines.
That is a matter of installing Docker and the runner (packages provided
by GitLab). Configuration is as simple as running one command and
pasting the URL as well as a token.
We can even go hybrid and use the best of both worlds. My tests suggest
that specific runners are preferred, using the shared runners as
fallback.

Let me know what you think
Jonas


1: https://lists.gnu.org/archive/html/lilypond-devel/2020-02/msg00416.html
2: Our 'lilypond' group is older than 
https://about.gitlab.com/releases/2020/03/18/ci-minutes-for-free-users/

signature.asc
Description: This is a digitally signed message part