Re: [lmi] Improving usability of automated tests

[Greg Chicares]

On 2017-02-08 16:28, Vadim Zeitlin wrote:
> On Wed, 8 Feb 2017 01:59:02 +0000 Greg Chicares <address@hidden> wrote:
> 
> GC> On 2017-01-13 23:30, Greg Chicares wrote:
> GC> [...]
> GC> > /opt/lmi/src/lmi[0]$make $coefficiency unit_tests >../log 2>&1
> GC> > /opt/lmi/src/lmi[0]$sed -f diagnostics.sed ../log 2>&1 |less -S
> GC> > [and searching for '\*\*' therein, excluding the test-framework test]
> GC> 
> GC> s/, excluding.*[]]/d
> GC> 
> GC> commit 453051202f5bd5d4683b825f0a7cf429445823e2
> GC> 
> GC> It may seem like only slight incremental progress, but it removes
> GC> a step that required manual activity and thought, and I run the
> GC> unit tests many times every day.
> 
>  Do I understand correctly that I can now check for the unit test failures
> just by searching for "***"? Or do I need to still filter out make errors
> using diagnostics.sed? If so, perhaps we could use another marker for lmi
> test failures, e.g. "!!!", to make it even easier?

Good point: 'make' uses a series of asterisks, so I'll cause these
tests to use something else instead.

I don't yet feel confident about depending on any simple filtering rule.
A test may...
 - succeed entirely
 - run to completion, but report errors
 - start running, but then abend
 - fail to compile
and I need to make sure I'm either
 - detecting absolutely all the abnormalities, or
 - count the successes and compare against an expected total.
But I'm trying not to spend any time on this yet, at least not until
we're happy enough with input sequences. (Otherwise, I would add that
maybe marking failures '????' and successes '!!!!' would be best;
but there I go again.)