Re: cunit [Was: Re: mailutils]

[Alain Magloire]

> 
> I'd like to see tests, and would really like there to be a
> test framework.

200% in agreement.

> Having one makes it easier and faster to write code.

Certainly would help catching stupid mistakes.

> The *unit test stuff (based on a 5 minute look
> at the cppunit docs :-)) are fancy asserts. Maybe nice, but
> certainly not necessary.

What do you suggest?

I don't know {j,cpp,c}unit very much,
however I know the reputation of Eric Gamma, which is a good sign.

Many GNU tools use dejagnu(tcl/tk) do not know much about it either.

> We write protocol toolkits at work, and a few of us have pushed
> hard to get comprehensive testing, and our experience is basically
> that untested code has bugs. No surprise there!

8-)

> To do this in mailutils I would want to
> 
> - fork a server that I will communicate with via a pipe

Possible, most of the daemon can be started in inetd mode
so if we make the setsockopt() a non fatal error in testing
mode they can work via a pipe.

> - run negative low-level tests, i.e. hard code a long pop3 dialog
>   with obscenely long commands to try and trigger buffer overflows
> - open mailboxes to talk to the server, and call our APIs with
>   both reasonable and totally unreasonable arguments
> 
> This would be easy, we just need two things:
> 
> - the ability to tell a server to run either chrooted, or to only access
>   config and mail spools in the test tree

would not the option --maildir do the trick?

> 
> - open a mailbox on arbitrary file descriptors (can we do this by
>   setting the stream after creating one?)

probably, we could create an arbitrary stream_t object that sends
garbage or good data intermix with garbage.

> 
> Also, memory leak testing is usually naive, it doesn't test the
> error paths, particularly whether memory is leaked in the cleanup
> paths triggered by a failed alloc. There are too many people running
> Insure++ saying "look, we don't have memory leaks",  but you take their
> code, start failing mallocs, and it usually doesn't leak, it explodes.


That's a good point.  The library itself try not to use things like xmalloc()
which simply exit() when malloc() failed but rather throw the error up and 
let the policy decision be made above.

> We can get close to 100% code coverage by running our tests in a loop:

[interesting bits .. deleted]
...
> IMO, one of the reasons a lot of servers have gone through their
> cycles of buffer overflows and other security problems is lack of
> any exhaustive testing. The standard test procedure, have people
> use it and see if it works, only tests the SUCCESS paths, people
> attacking a system are going to try and exploit the failure paths.


200% agree.  I'm ready to spend the few cycles
if you can point to me at an interesting framework for tests
[cunit] ?