Re: [lmi] Improving usability of automated tests

[Vadim Zeitlin]

On Fri, 13 Jan 2017 23:30:58 +0000 Greg Chicares <address@hidden> wrote:

GC> On 2017-01-13 16:49, Vadim Zeitlin wrote:
GC> [...]
GC> >  The fix is to either revert these changes or, IMHO preferably, add our 
own
GC> > trivial begin()/end() implementations. The attached patch does the latter
GC> > and does pass the test suite now, but please let me know if you'd prefer 
to
GC> > be more conservative and revert these changes instead.
GC> 
GC> Yes, that's what I prefer. Let's change to ranged for-loops only where
GC> the changes are "simple".

 I must admit I quite liked iterating over the directory using a short
loop, but I do have to agree that it's not totally obvious and simple.

GC> >  And sorry again for somehow not noticing the test failures :-( I know 
it's
GC> > entirely my fault for not being attentive enough, but I'd still like to
GC> > note that it's too easy to not see the failures in the huge amount of
GC> > output from "make unit_tests". Compare this with the IMHO much nicer 
output
GC> > from Automake-generated "make check" target which looks like this:
GC> > 
GC> > 
============================================================================
GC> > Testsuite summary for LMI 1.0.0
GC> > 
============================================================================
GC> > # TOTAL: 63
GC> > # PASS:  58
GC> > # SKIP:  0
GC> > # XFAIL: 0
GC> > # FAIL:  5
GC> > # XPASS: 0
GC> > # ERROR: 0
GC> > 
GC> > (it's even coloured, with both "FAIL" and the header highlighted in red)
GC> > and, of course, "make check" exits with non-zero exit status in this case.
GC> 
GC> Similarly, and with no better excuse really, I avoid running the
GC> automated GUI test, merely because I don't like to clean up after it.
GC> It leaves files like
GC>   ABC.John_Brown.000000001.monthly_trace.tsv
GC>   testfile.mec.xml
GC>   unnamed.gpt.xml
GC> in [chroot]/opt/lmi/bin ; worse, it overwrites 'configurable_settings.xml',
GC> so that any careful choice of calculation-summary columns is wiped out
GC> unless I had thought to save it elsewhere, which I often omit to do.

 This is indeed quite bad, but, IIRC, it was a requirement that I
dutifully, if not cheerfully, implemented, wasn't it?

GC> Let's make a pact: once this ranged-for-loop logjam is cleared, I'll make
GC> the 'unit_tests' target print a summary if you'll make the GUI test clean
GC> up its accidental artifacts. And then we'll both run all these tests more
GC> often.

 I'll definitely clean up the GUI test. But I actually hoped to avoid
creating more work for you by making sure I can run the unit tests using
automake instead, there is not that much to do to make it happen.
Unfortunately it will still require some changes to the sources themselves,
i.e. I can't fix this purely at automake level. I am not ready to propose
these changes yet, but I've at least described the current problems at
https://github.com/vadz/lmi/issues/53 which can be useful if you ever want
to run the unit tests in the native Unix build (some of these problems
probably won't happen when using official makefiles, but at least (5) and
(6) definitely will).

GC> But this is my lucky day (and 1956-1-13 was a Friday the thirteenth
GC> as well). The one thing I miss about using an msw VM is that I could
GC>  - type the command to run the GUI tests, but not press Enter
GC>  - move the mouse pointer outside the VM window, but not click
GC>  - and then press Enter and click the mouse
GC> and it would just work--the tests would run as though they were on a
GC> separate machine, and I could do something useful elsewhere instead of
GC> watching them run.

 Yes, VMs are very convenient for automated GUI testing. I wonder if you
could run Wine in an Xnest to come closer to VM experience?

VZ