[bug#60788] Policy for system tests?

[Ludovic Courtès]

Hi,

Maxim Cournoyer <maxim.cournoyer@gmail.com> skribis:

> Ludovic Courtès <ludo@gnu.org> writes:

[...]

>> I think a system test would be nice, we generally require it upfront,
>> but since Maxim wrote it can come later, let’s not let it block this
>> patch any longer.
>
> I didn't mean to lower our standards; I wasn't sure if that policy of
> ours was strict, since a few system services do not have corresponding
> tests, IIRC.  I wary a bit that demanding a system test for each added
> service may cause scalability problems in the long run, as each demand a
> disk-heavy image to be generated and the test to run in a VM, which
> makes it expensive/slow.  On the other hand, it's nice to know about any
> regressions when they happen rather than on a reboot...

Yeah, it’s an unwritten policy; I think we’ve consistently required it
for some time now.  It’s useful because otherwise it’s hard to tell what
the status is for a service.

(Speaking of which, we do *not* have that policy for Home services,
because we don’t even have a test strategy, and that’s something we
should fix before it’s too late.)

> If we have such a policy, perhaps we should explicit it in
> our documented contribution guidelines?

Yes!

Also, we should split the submission guidelines into different
categories: packages, services, doc, core, etc.

Ludo’.