[RFC] Automatic 'make check' in CI

[Jonas Hahnfeld]

Hi all,

I'd like to present a first workable version of 'make check' for use in
our CI pipelines. I've pushed the necessary commits to my personal fork
and created two merge requests to demonstrate the results:

1. Run 'make check' for merge requests (no difference)
URL: https://gitlab.com/hahnjo/lilypond/-/merge_requests/5
Job: https://gitlab.com/hahnjo/lilypond/-/jobs/858498690
test-results:
https://hahnjo.gitlab.io/-/lilypond/-/jobs/858498690/artifacts/test-results/index.html

2. Introduce difference in bar-lines.ly
URL: https://gitlab.com/hahnjo/lilypond/-/merge_requests/6
Job: https://gitlab.com/hahnjo/lilypond/-/jobs/852618720
test-results:
https://hahnjo.gitlab.io/-/lilypond/-/jobs/852618720/artifacts/test-results/index.html

This first workable implementation contains the minimum functionality:
It runs 'make test-baseline' for every push to the master branch and
replaces 'make test' in the pipelines for MRs with 'make check'. The
test-results are uploaded as artifacts and can be either downloaded as
zip archive or viewed directly (see above).

There are a few known issues that I'm aware of:
 - I needed to delete input/regression/option-help.ly because it logs
the options currently in use by lilypond, including the job-count which
varies when using our own runners with more than one core.
 - The test-results are pretty hard to find: Even if you know where to
look, it takes at least three clicks to download the zip archive plus
extracting and opening index.html; viewing the results directly takes
another three clicks from the job page (via 'Browse'). To ease the
second option, I've included a link in the latest jobs of !5 above. The
link is clickable which is a new feature of GitLab that I asked to be
enabled for my repo, so it won't work directly for lilypond/lilypond.
 - The gittree information does not contain the right merge base. This
is related to issue #6061 and I've submitted a change to GitLab to
expose the needed information in CI, but it wasn't merged yet.
 - The modified job for MRs always downloads the latest test-baseline
from master and not the one corresponding to the merge base. Same
reason as the previous point, I hope we can get the needed information
soon-ish. Until then, we need to rebase to adapt the MR commits to the
available test-baseline.
 - Sometimes the test-results contain spurious diffs, for example here:
https://hahnjo.gitlab.io/-/lilypond/-/jobs/858441670/artifacts/test-results/index.html
I can reproduce this locally with --enable-checking, but haven't
investigated further yet (there were a couple of other problems that I
needed to solve in order to get things working...). If somebody has an
idea for a fix, that would be great but I think these can be safely
ignored for now.

There are a few more elaborate things that I'd like to work on in the
future. For example, GitLab can show a list of 'failing' tests which
can tell us at first glance if we need to look into the test-results.
I've prototyped this integration in the second MR, but it's very
misleading because the file extensions are missing and GitLab prints "0
failed out of null" when there are no tests. The obvious solution is to
mark all existing tests as success, but this requires a bit more
thought to integrate into output-distance.py (or somewhere else).

Despite these shortcomings, I think it would make sense to enable this
first implementation in lilypond/lilypond. What do you think?

Jonas


P.S. I probably forgot a few important things; please ask if some
details are not clear...

signature.asc
Description: This is a digitally signed message part