[lmi] Better unit tests

[Vadim Zeitlin]

On Thu, 23 Aug 2018 19:52:57 +0000 Greg Chicares <address@hidden> wrote:

GC> On 2018-08-23 18:20, Vadim Zeitlin wrote:
[...]
GC> >  I wonder if there is any chance of convincing you to use it instead of 
the
GC> > current test_tools.hpp? I'd be very enthusiastic about this...
GC> 
GC> You might be unaware of the coding horrors that lurk in some of those
GC> old unit tests. Not long ago, I rewrote the ancient input_sequence
GC> tests, and found that I could hardly understand them. It would be
GC> easy to replace a test that's valid and useful (but comprehensible
GC> only with great difficulty) with one that is easy to read but checks
GC> something rather different and less valuable. To do this well would
GC> be extremely costly.

 I don't see how could this happen, to be honest. I'm speaking of
more-or-less mechanically replacing BOOST_TEST() and similar macros with
CHECK() (or CATCH_CHECK() if you prefer being explicit) etc. In principle,
we could even simply redefine the existing BOOST_TEST() in terms of CHECK()
but this is not going to help with confusion...

GC> Besides, my test-running experience is already nearly perfect. I run
GC>   $./nychthemeral_test.sh 2>&1 | tee /tmp/lmi/logs/log | sed -f errors.sed
GC> often dozens of times a day, and certainly before every git-push.

 So should I be running this one here too instead of just running "make
unit_tests" as I currently do? I'm not sure if I have everything the
nychthemeral_test needs.

GC> Can you name some features that I might find useful? I've read
GC>   https://github.com/catchorg/Catch2/blob/master/docs/why-catch.md#top
GC> line by line, but found nothing to draw my interest.

 I think this assertion merits its own separate email thread, so I've
started this message to list things that I find useful (this is not meant
to be an exhaustive list):

- Much better error reporting in case of test failure by default, without
  any extra effort.

- In particular, no need for the unnatural syntax of BOOST_TEST_EQUAL(),
  just CHECK(x == y) works perfectly fine.

- No need to explicitly run all the test functions. You just define them
  and they will be run automatically (and, if desired, in random order).

- No need to write main(). You can, if you want to, but the default one
  is usually just fine.

- This is a huge one for me: possibility to run a single test, or a single
  subtest of a test from the command line: you don't need to always rerun
  all of the tests, some of which can be time-consuming, or, when running
  under debugger, trigger a breakpoint you're interested in hundreds of
  times before the one you're interested in. You can just specify the test
  you want to run/debug directly.

- BTW, speaking of subtests (called sections in CATCH), they're a much more
  C++-ish equivalent to Java/OOP test fixtures.

- Built-in support for timing the tests. This may be not as precise as
  TimeAnAliquot(), but doesn't require doing anything at all and is quite
  handy to check if there are any obvious regressions after a change.


 Frankly, CATCH is the most frictionless way of writing tests in C++ I've
seen so far and I can't imagine how could a testing library be simpler than
that, while still remaining useful. It's much easier to list its
disadvantages than its advantages because, after using it for a few years,
I still know of only one: compilation speed. It is pretty slow, especially
for small tests, where the compile-time of catch.hpp can easily dominate
the compile-time of the test itself. However if you merged all lmi tests in
a single unit test program (something that I'd definitely love
independently of CATCH, as I could then build it easily with MSVS, while
creating hundreds of projects for the individual tests is anything but
easy), which wouldn't be impractical if you could still run just some of
the tests, as it can be done with CATCH, it would still build faster than
all the individual tests right now. And if compilation speed is really a
problem, then you might consider one of CATCH forks, that preserve all the
advantages above, but compile much faster, such as lest or doctest.


 FWIW, when I examine some C++ library, I always look at its tests first.
If it doesn't have any, I don't look any further. If it uses some
xUnit-inspired test framework, I pinch my nose and look at it if I really
have to. If it uses Boost.Test or GTest, I check whether it was updated any
in this decade (and usually it wasn't). If it uses CATCH (or one of its
forks/imitators), I start looking at the documentation and all the rest...

VZ