[lmi] Improving usability of automated tests [Was: [PATCH] C++ m11n: range-based for loops]

[Greg Chicares]

On 2017-01-13 16:49, Vadim Zeitlin wrote:
[...]
>  The fix is to either revert these changes or, IMHO preferably, add our own
> trivial begin()/end() implementations. The attached patch does the latter
> and does pass the test suite now, but please let me know if you'd prefer to
> be more conservative and revert these changes instead.

Yes, that's what I prefer. Let's change to ranged for-loops only where
the changes are "simple".

>  And sorry again for somehow not noticing the test failures :-( I know it's
> entirely my fault for not being attentive enough, but I'd still like to
> note that it's too easy to not see the failures in the huge amount of
> output from "make unit_tests". Compare this with the IMHO much nicer output
> from Automake-generated "make check" target which looks like this:
> 
> ============================================================================
> Testsuite summary for LMI 1.0.0
> ============================================================================
> # TOTAL: 63
> # PASS:  58
> # SKIP:  0
> # XFAIL: 0
> # FAIL:  5
> # XPASS: 0
> # ERROR: 0
> 
> (it's even coloured, with both "FAIL" and the header highlighted in red)
> and, of course, "make check" exits with non-zero exit status in this case.

Similarly, and with no better excuse really, I avoid running the
automated GUI test, merely because I don't like to clean up after it.
It leaves files like
  ABC.John_Brown.000000001.monthly_trace.tsv
  testfile.mec.xml
  unnamed.gpt.xml
in [chroot]/opt/lmi/bin ; worse, it overwrites 'configurable_settings.xml',
so that any careful choice of calculation-summary columns is wiped out
unless I had thought to save it elsewhere, which I often omit to do.

Let's make a pact: once this ranged-for-loop logjam is cleared, I'll make
the 'unit_tests' target print a summary if you'll make the GUI test clean
up its accidental artifacts. And then we'll both run all these tests more
often.

Reading between the lines, you may already have discerned that I've been
doing

/opt/lmi/src/lmi[0]$make $coefficiency install check_physical_closure 2>&1 
|less -S
/opt/lmi/src/lmi[0]$make $coefficiency cgi_tests cli_tests 2>&1 |less -S
/opt/lmi/src/lmi[0]$make $coefficiency system_test 2>&1 |less -S
/opt/lmi/src/lmi[0]$make $coefficiency unit_tests >../log 2>&1
/opt/lmi/src/lmi[0]$sed -f diagnostics.sed ../log 2>&1 |less -S
[and searching for '\*\*' therein, excluding the test-framework test]

all day, but haven't run the GUI test until just now. I had run it
the other day, and I'm not concerned about this error that I saw then:

about_dialog_version: ERROR (Assertion '(version_year) == (copyright_year)' 
failed (expected 2017 vs observed 2016). [file 
/opt/lmi/src/lmi/wx_test_about_version.cpp, line 189] )

which merely suggests that I should 'make release_candidate' right
after updating copyright notices. I was however frustrated to see:

calculation_summary: ERROR (Assertion failure: Expected preferences dialog was 
not shown. [file /opt/lmi/src/lmi/wx_test_calculation_summary.cpp, line 100, in 
run()].)
...
paste_census: ERROR (Assertion 'dvc' failed. [file 
/opt/lmi/src/lmi/wx_test_paste_census.cpp, line 69] )
pdf_census: started
...
FAILURE: 18 out of 25 tests failed.
NOTE: 4 tests were skipped

and how can I take you to task for missing two unit-test failures
when I omitted to run the GUI test because I dislike its side effects,
thereby apparently setting myself up for another day of debugging?

But this is my lucky day (and 1956-1-13 was a Friday the thirteenth
as well). The one thing I miss about using an msw VM is that I could
 - type the command to run the GUI tests, but not press Enter
 - move the mouse pointer outside the VM window, but not click
 - and then press Enter and click the mouse
and it would just work--the tests would run as though they were on a
separate machine, and I could do something useful elsewhere instead of
watching them run.

I can now say definitively that a chroot does not behave that way,
and neither does an xfce workspace. Repeating the steps above through
"press Enter" and then touching nothing until the tests finish, all
tests succeed except the expected version_year one:

about_dialog_version: ERROR (Assertion '(version_year) == (copyright_year)' 
failed (expected 2017 vs observed 2016). [file 
/opt/lmi/src/lmi/wx_test_about_version.cpp, line 189] )
FAILURE: 1 out of 25 tests failed.