bug#59317: 29.0.50; Feature idea: suppress `message' output in ert batch test output

[Matt Armstrong]

Michael Albinus <michael.albinus@gmx.de> writes:

> Stefan Kangas <stefankangas@gmail.com> writes:
>
> Hi Stefan,
>>> Besides the other arguments already given in this thread, I'd like to
>>> emphasize that messages are useful even in batch mode, for successful
>>> tests. This is the only way to study ert tests in batch mode, for
>>> example on our CI/CD machines hydra and emba.
>>
>> It would be less than useful in CI, indeed.
>>
>> But in interactive sessions, and when running tests manually with
>> e.g. M-x compile, it helps to have less noise, to find the relevant
>> failures faster.
>
> Maybe, although the messages in interactive mode go to a special *ERT
> Messages* buffer.

I believe Stefan's idea as running something like "make check" through
M-x compile, so the tests run in batch mode with output in *Compilation*
buffers.

As an implementation note, the *ERT Messages* buffer is not where
`message' output goes during a test.  Instead, the `ert-test' function
notes the `point-max-marker' of the *Messages* buffer, then runs the
test, and then extracts with `buffer-substring' the region of *Messages*
between that marker and its (point-max).  It then squirrels that string
away in a test result data structure.  Notably, ert doesn't suppress
message output in any way during a test.  They still appear in the
minibuffer, go to *Messages*, etc.


> Btw, there are test cases which depend on proper messages. See for
> example autorevert-tests.el. For those tests, messages must not be
> suppressed at all.

The way I imagine it, any lisp level introspection, mocking, etc., would
still work.


>>> If messages shall be suppressed, this must be controlled by a user
>>> option. However, I'm not in favor to do it at all.
>>
>> Agreed.  Perhaps an environment variable could also be provided, so
>> that it is easier to control from the command line?
>
> Sure. There are already several $EMACS_TEST_* environment variables,
> see test/README.

Good to know.