Re: test setup advice

[Marc Alff]

Hi Tom,


tom fogal wrote:

> I'm looking for advice on how to structure a build system using
> automake.
>
> Thanks for any solutions / ideas / comments.
>
> -tom
>  

These are just random thoughts / ideas that I used on C++ projects,
just my two cents (and in all cases, my personal opinion, your mileage 
may vary).
It's not automake specific and more high level, but I implemented these 
with automake and it worked well.

Any developer will use an excuse that tests are slow to build / 
difficult to maintain to actually
not maintain them, so being sensitive to that is very healthy. However, 
it's not limited to tests
alone but true for the entire code base, so having code that builds fast 
and lean improves development
in general.
To achieve that, a couple of techniques do work in C++ :
- try to limit the #includes dependency hell. Sometimes a forward (class 
foo;) is much better that actually
including foo.h
- avoid including /usr/include/xxx.h (C stdio, C++ stream, etc)  in 
header files, include that in .cpp files instead.
- once in a while review the dependencies generated by automake on 
header files : it's very instructive
and shows you what the compiler has to deal with. I personally prefer to 
cut dependencies rather that using
pre-compiled headers of other things like that.
- templates are powerful, but can become quickly hell when abused.

Tests do influence design : for the production code itself, the better 
it's designed, the easier it is to test.
Writing tests at the same time of the production code actually 
influences the code to a better karma.
Writing tests after the facts on spaghetti code is just damage control, 
not to mention it's very difficult.

Separating "production" and "test" code in different directories is very 
healthy, for many reasons :
- all the test code can be instantly unplugged to make sure that the 
production code builds without using
test code while compiling against dummy header files, or worst linking 
against test stubs.
- porting your code to a different platform can be done for the 
production code alone if needed,
in case the test code uses tools not available everywhere.
- looking from the CVS point of view, someone checking in a change in 
production code makes me more
nervous that someone checking in a new test, and having a simple way to 
find out which code is which
really helps.

On test themselves, I make the distinction between :
- test drivers (code that you actually have to write for a new family of 
test cases),
- test cases (just a description of the test itself, might not be C++ 
code but just a text or xml file).

The idea is to write a test driver that reads an external script that 
tells it what test to execute,
or what function to call with what value. This code is typically more 
complex to write, but is written once.
A test case (typically a script) is when written for each test.
Adding a test case does not involves any build, and can be done at any time.
It's also very useful for debugging, since anyone (including non 
programmers) can write a new case at anytime
(provided of course that it uses the feature of existing drivers).

If each test consists of code then yes, it takes time to build, and 
effort to maintain.
If each test case consist of a script that talks to a test driver, 
building is very fast,
maintenance can be a breeze of hell, depending on the design.
A word of warning, good planning is the key there.

In general, I prefer "small" test binaries and not linking to 
everything.a, so that once the test
gives the correct result, the test driver can be leveraged to do :
- performance profiling (gprof),
- code coverage (tcov)
- or dynamic analysis (purify like tools).
The idea is that since someone made an effort to write it (it's 
automated in the makefile, right ?),
it might as well be used to it's full value.

tcov is very important, since a good test suite will try to improve 
coverage,
not beat the same function to death while ignoring all the others,
and it's very hard to access coverage without a tool (take a wild guess, 
divide by 10, that's the real coverage).

That's all for now, sorry if it's a bit off list.

I hope this helps with ideas.

Cheers,
Marc Alff.