Re: unit tests best practice

[Alex Sassmannshausen]

Hi Cato and Mike,

Thanks for the discussion — this is already shedding a ton of light :-)

Catonano writes:

> Il giorno gio 3 gen 2019 alle ore 19:45 Mike Gran <address@hidden> ha
> scritto:
>
>> On Thu, Jan 03, 2019 at 05:53:33PM +0100, Catonano wrote:
>> > Wat's the best practice to instrument a Guile based project for running
>> > unit tests ?
>> >
>> > guile-git has this fragment in the Makefile.am file
>> >
>> > SCM_LOG_DRIVER =                                \
>> >   $(top_builddir)/pre-inst-env                  \
>> >   $(GUILE) --no-auto-compile -e main            \
>> >       $(top_srcdir)/build-aux/test-driver.scm
>> >
>> > as you can see, what Automake produces calls a "main" procedure in a
>> > "test-driver.scm" script in the "build-aux" folder
>> >
>> > The effect is that you can call
>> >
>> > make check
>> >
>> > in the root folder and have the tests run
>> >
>> > Instead, in the Makefile.am file produced by guile-hall wired projects
>> you
>> > ave this fragment
>> >
>> > AM_TESTS_ENVIRONMENT = abs_top_srcdir=\"$(abs_top_srcdir)\"
>> > SCM_LOG_COMPILER = $(top_builddir)/test-env $(GUILE)
>> > AM_SCM_LOG_FLAGS = --no-auto-compile -L \"$(top_srcdir)\"
>> >
>> > so there's a SCM_LOG_COMPILER instead of a SCM_LOG_DRIVER
>> >
>> > What is this LOG_COMPILER doing, exactly ?
>>
>> OK, this is a muddle, but, it goes like this.
>>
>>
> Mike,
>
> thank you for taking the time to clarify

Indeed, this is very helpful.

> I think the less painful step at this stage would be to make guile-hall
> arrange the tests in the same way that guile-git does

So, currently, guile-hall does as Mike did: it uses LOG_COMPILER and
expects the test scripts to provide an exit code.

A simple but probably bad way to achieve that is to add:
(exit (= (test-runner-fail-count (test-runner-current)) 0))

to the bottom of srfi-64 test files.  The reason why it might be bad is
that it does not generate nice exit codes, but rather simply non-zero if
any test fails.

I would be happy to have guile-hall support LOG_DRIVER instead/as well.

The best way to achieve that would be to have a template driver script,
and appropriate changes to the Makefile.am generator.

We could then default to TEST_DRIVER, but provide a command line flag to
support LOG_COMPILER.

> Right now it's the other way around but I don't think Alex would complain,
> he openly declares his cluelessness in the matter

Quite.

> In this way, guile-hall could provide a working solution for new projects
> without too much work in wrapping SRFFIs in order to provide compliance
> with Automake

Agreed, TEST_DRIVER would probably be closer to what people would expect.

> The wrapping could me made in a further step

Also agreed.

Alex