Re: new procedure with GitLab CI

[James Lowe]

On 24/05/2020 08:59, Jonas Hahnfeld wrote:
> Hi James,
>
> Am Samstag, den 23.05.2020, 19:08 +0100 schrieb James Lowe:
> > Jonas,
> >
> > On 23/05/2020 18:59, Jonas Hahnfeld wrote:
> > > Hi all,
> > >
> > > I've now made the necessary settings, merged the changes in
> > > https://gitlab.com/lilypond/lilypond/-/merge_requests/57, changed all
> > > existing merge requests to target 'master', and deleted 'staging'.
> > > I've not rebased the existing merge requests and there's no need to do
> > > so if it's already in one of the later stages (you'll be forced on
> > > submission). However remember that James won't get MRs with Patch::new
> > > for manual regression testing unless it has passed automated tests.
> I just noticed that I forgot to merge the change to countdown.py:
> https://gitlab.com/lilypond/infrastructure/-/merge_requests/5
>
> The renamed flag --testing now only shows patches in Patch::new that
> have passed CI testing. There's no (automatic) infrastructure yet to
> reset failed patch to Patch::needs_work, I'll play bot until this
> becomes a reality.
>
> > So what do I continue to/stop doing manually?
> >
> > Sorry if it isn't that clear for me, but is this replacing my testing or
> > just patchy-staging merge?
> GitLab CI is replacing patchy-staging entirely. With respect to your
> process, it does not (yet) run 'make check' for regression testing.
> This would need to continue happening in a manual fashion as before.
> However you don't need to run 'make doc' and you don't need to test
> patches that failed automatic testing (see above).
>
> Jonas

OK. Using Masamichi's MR as an example (nothing personal Hosoda-san!) I 
saw that his MR !81 came up via countdown.py - I am using the latest 
update of this BTW - so looking at this MR via the web I could not tell 
if this had done its make doc or not.

A few mins later I saw your email/update in the ticket to Hosoda-san and 
and I can also see a new commit/MR update in the same thread and that 
!81 still appears in the countdown.py list.

So has this done a make doc or not? I am still unsure at this point.

See:https://gitlab.com/lilypond/lilypond/pipelines (as of 11:25BST anyway)

It has (had) no mention of !81 and I cannot see anything in the thread 
that says it has passed make doc.

So now I am unsure what to do, because if I run my new set of tests I 
*won't* be doing make doc, and if I change this MR to Patch::review and 
it hasn't had a make doc, then it will go through the countdown not 
tested properly right?


Q1. What cases will I get a false positive with countdown.py if a MR has 
not had a make doc done - this seems, at the moment to be user-error prone.

Q2. How can I be sure (at least for the first few dozen or so patch 
tests) that the MR really has done the make doc via CI? (i.e. without 
the script) should I see the MR in that URL I listed above?

Q3. How does user know that his patch has failed the make doc (and that 
I won't have even tested it) I assume this is because the patch will get 
set back to 'needs_work' (and they'll get notified)?

Thanks as always Jonas