[Gcl-devel] ANSI compliance in gcl

[Paul Dietz]

Walt,

  I've been working on testing gcl for ansi compliance,
with the notion that establishing where gcl is noncompliant,
and providing tests for this noncompliance, is a necessary
first step to achieving compliance (a 'test-first' strategy
such as is used in Extreme Programming).

  My mode of operation has been to go through the ANSI CL
Hyperspec and convert the specifications into test cases.
I've done the following parts of the spec:  most of section 3,
part of section 4, all of sections 5 and 6, a little of section
8 (I am working on complete tests for DEFSTRUCT right now),
all of sections 10, 11, 13, and 14, part of section 15,
and all of sections 16 and 17.  Other parts have been
incidentally tested (in that functions from those parts
are used to implement the other tests), but not in any
rigorous way.

  I use a hacked version of Waters' RL test package to
run the tests.

  These tests are intended to be ANSI CL compliance tests,
not regression tests for gcl (although if bugs are found in gcl
that can be tested by a test that must work in any ANSI compliant
lisp I do not hestitate to add it).  As such, they must be designed
so that they must pass in *any* compliant implementation of
ANSI CL.  It is very useful to run the tests through other
implementations to check for this -- I use Allegro CL and
CMUCL; others run the tests through CLISP and SBCL (and I
think Lispworks).  I know the CMUCL and SBCL teams are using
the tests to fix compliance corner cases in their implementations.
We often find bad tests this way.  In some cases, the spec
is ambiguous and it is unclear just what is required.  At that
point there's discussion in cmucl-imp, comp.lang.lisp, and
elsewhere to try to figure out just what the spec calls for.

  I have deliberately not copied existing CL test suites
(there is one available on CLOCC, for example).  I've done
this for two reasons.  First, those test suites are not necessarily
just ANSI CL compliance tests, in the sense that all tests
must pass in any compliant implementation.  Second, I've found
that writing the tests myself from the spec has led to additional
problems being found in other lisps.  I assume those test suites
have been run on those lisps, so they must have holes in their
test coverage.

  As I wrote above, I'm working on the DEFSTRUCT tests now, so
please don't update them.  If you'd like to review existing tests,
or begin work on tests for one of the other uncovered parts of the hyperspec,
go right ahead.  I am trying to be very thorough here; we've got
nearly 6000 tests so far; I estimate we'll have perhaps 15000 tests
when we're done.

        Paul Dietz
        address@hidden