Re: [lwip-devel] testing methods

[Kieran Mansley]

On Tue, 2007-03-13 at 17:01 -0500, Taranowski, Thomas (SWCOE) wrote:
> At the risk of being drug into the street and mauled by angry
> developers, I’d like to know if we have any sort of informal test
> framework for the lwIP stack.

We have none.  It is a shambles and embarrassing, but despite that lwIP
seems to be pretty stable for most users.

> 
> Now, I’m currently working on developing a test suite for my own
> needs, which will be fairly comprehensive, with respect to the BSD
> socket layer.  I’d like to propose to my management that we release
> this to the lwIP community for further enhancement and support.

My doubts here would be that:
 - we already know that the lwIP sockets layer is not particularly
compliant with the standard, and doesn't support a lot of standard
operations.
 - it's hard to exercise large portions of the code using the sockets
API alone.

A comprehensive socket API tester is also many man years of work!  

> The primary reason for doing this is because I think it will result in
> a better product, but another reason, perhaps more compelling for
> some, is because we’re all doing our own testing thing, and it would
> be more efficient if we could collaborate on a shared test
> framework.  
>
> Before I work up my proposal, I’d like to know if we already have
> something, and if anyone else thinks this would be a useful addition
> to our lwIP project.

Thanks for taking the time to involve lwIP - it would clearly be to our
benefit.

There are some existing commercial test suites for TCP stacks.  ANVL is
one I've used in the past (which although useful, has a number of bugs
in its tests).  They don't come cheap though.

Kieran