Re: cunit [Was: Re: mailutils]

[Sam Roberts]

I'd like to see tests, and would really like there to be a
test framework. Having one makes it easier and faster to
write code. The *unit test stuff (based on a 5 minute look
at the cppunit docs :-)) are fancy asserts. Maybe nice, but
certainly not necessary.

We write protocol toolkits at work, and a few of us have pushed
hard to get comprehensive testing, and our experience is basically
that untested code has bugs. No surprise there!

To do this in mailutils I would want to

- fork a server that I will communicate with via a pipe
- run negative low-level tests, i.e. hard code a long pop3 dialog
  with obscenely long commands to try and trigger buffer overflows
- open mailboxes to talk to the server, and call our APIs with
  both reasonable and totally unreasonable arguments

This would be easy, we just need two things:

- the ability to tell a server to run either chrooted, or to only access
  config and mail spools in the test tree

- open a mailbox on arbitrary file descriptors (can we do this by
  setting the stream after creating one?)

Also, memory leak testing is usually naive, it doesn't test the
error paths, particularly whether memory is leaked in the cleanup
paths triggered by a failed alloc. There are too many people running
Insure++ saying "look, we don't have memory leaks",  but you take their
code, start failing mallocs, and it usually doesn't leak, it explodes.

We can get close to 100% code coverage by running our tests in a loop:

for failon = 1 ...
do
   tst_FailNthMalloc(failon)

   status = Test

   if (tst_MallocFailHit())
      assert(status == ENOMEM)
             // if malloc failed this must be
             // indicated by the return codes, getting some
             // other error (like "parse failed") is WRONG.
   else
      assert(status == EOK)
             // otherwise the Test must have succeeded, possibly
             // by failing in the right way if its a negative test
done

This is easier for us because we don't call malloc directly, the user
provides the implementation through a callback, it might be possible
with gcc linker magic.

100% code coverage doesn't guarantee you don't have bugs, of course,
but less than 100% guarantees there is code that has never been
run. From trying to bug embedded systems that have intermittent and
unreproduceable bugs, I'm pretty convinced that a lot of those are
caused by "it will never happen" secenarios that do happen, just not
very often.

IMO, one of the reasons a lot of servers have gone through their
cycles of buffer overflows and other security problems is lack of
any exhaustive testing. The standard test procedure, have people
use it and see if it works, only tests the SUCCESS paths, people
attacking a system are going to try and exploit the failure paths.

The recent insecurities in SNMP were discovered by implementing
a well designed test suite.

Sam

Quoting Richard Dawe <address@hidden>, who wrote:
> Hello.
> 
> Alain Magloire wrote:
> [snip]
> > Is there a cunit?
> [snip]
> 
> Yup:
> 
>     http://cunit.sourceforge.net/

-- 
Sam Roberts <address@hidden> (Vivez sans temps mort!)