Re: improving basic tests

[Paul Kienzle]

On Mar 9, 2004, at 8:59 PM, John W. Eaton wrote:

> We could use some improved tests for basic operations in Octave.  For
> example, I discovered several bugs in matrix indexing in the last few
> days that could have been found long ago if we had a more
> comprehensive set of tests for indexing.  It would be very helpful if
> someone (or some group) could volunteer to work on this task.

Once the infrastructure is in place, this work is highly
distributable.  It's also an excellent way to contribute
back to the community.

> The current set of tests that are run by "make check" when Octave is
> built from sources are far from complete, and are not in a form that
> is convenient for running outside of the build environment.  It would
> be better to have a set of tests that could be run through either
> Octave or the other leading brand so we will have a way to do
> automated compatibility testing.

Definitely valuable.  In fact the bulk of my testing work on
octave-forge came when I was converting my signal processing
routines for use on a machine which did not have octave
(or the personal disk space for me to compile it).

> For example, we need someone to write tests of the form
>
>   x = 1;
>   check (x(zeros (1, 0)) == zeros (1, 0));
>   check (x(zeros (0, 1)) == zeros (0, 1));
>   ...
>
> for various object types and index types, where check is a function
> that reports test failures.

I call it assert, and I have it take three parameters: observed, 
expected
and tolerance (+ for absolute, - for relative).  For the most part I use
absolute tolerances because relative tolerances fail so badly near zero.
One little-known feature which improves the quality of assert is the
predefined variable argn, with which you can include the actual
argument text in your error message.

I do have a problem with assert in that it is not type extensible.  The
only way to assert conditions on a new type is to change the type
to one of the builtin types, or to modify assert to handle the new type.
It may work, however, to do:
        feval(['assert_',strrep(typeinfo(x),' ','_')], x, y, tolerance)
then for example the complex sparse type could supply
        assert_complex_sparse
The function isequal could be similarly defined.

I also have the function fail which expects a string expression
to fail with a certain error message pattern.

> The above code is just a simple example.  We will probably want to use
> and build on the testing code that Paul Kienzle has written for the
> octave-forge project, but we need to write a set of tests for basic
> built-in operations, and I think his work has mostly focused on tests
> for functions.

I rewrote the tests for sparse using my framework.   Or more correctly, 
I
wrote a shell script to produce the tests for sparse so that I could
test all combinations of scalar/full/sparse real/complex operations.
This involved mostly operator testing, but some indexing tests as well.
Automatic test generation is certainly the way to go for a lot of things
you want to do, though shell is not necessarily the best tool for the 
job.

Paul Kienzle
address@hidden



-------------------------------------------------------------
Octave is freely available under the terms of the GNU GPL.

Octave's home on the web:  http://www.octave.org
How to fund new projects:  http://www.octave.org/funding.html
Subscription information:  http://www.octave.org/archive.html
-------------------------------------------------------------