Re: testing framework and package.el

[Christian Ohler]

 On 12/10/10 14:02, Stephen J. Turnbull wrote:
> Christian Ohler writes:
>
>   >  AIUI, XEmacs has its tests in a tests/ directory next to lisp/, in the
>   >  same repository.  Are you saying this has proven to be a bad idea?
>
> Not bad; suboptimal.  There are two kinds of tests, tests of the LISP
> language (including Emacs extensions described in the Lispref), and
> tests of the internal implementation.
>
> The former should be used for all versions of X?Emacs Lisp, and
> therefore are conceptually separate from the sources.  They often need
> to be versioned, but usually a boundp or fboundp test is good enough
> for that.
>
> The latter, of course, is implementation-specific and tied to the
> sources.
>
> In XEmacs practice, the two are mixed together, and as bugs are fixed
> and features are added the tests for different versions have diverged,
> even where the features are the same.  This means that, for example,
> it's possible (and not uncommon, in fact) for 21.4 to be missing
> regression tests that are present in 21.5 for 21.4 features.

I see, thanks for clarifying.  Is it really mainly ELisp language 
features where this applies?  Any test that I write now could make sense 
for the previous Emacs version, or across all Emacsen, if it's a shared 
feature.  Probably we don't care about compatibility or regressions of 
specific UI features too much, so sharing such tests is perhaps not 
worth the trouble.  Still, do you think there's a strict boundary 
between shared tests and implementation-specific tests, or a continuum?  
If it's a continuum, perhaps `ert-deftest' should have a :since keyword 
that takes some kind of structured value that specified the Emacs 
versions that are supposed to support the feature being tested.  OTOH, 
if tests really mostly fall in one of two categories, a simpler solution 
would be sufficient.

>   >  a way that these actions are as easy as possible.  I expect that
>   >  running the current tests against older Emacs versions is something
>   >  that we will want to do much less frequently, so we should not
>   >  optimize for this use case at the expense of the others.
>
> What's so hard about prepending "../" to a few Makefile variables?

I'm not sure I understand your question.  I was trying to say that it 
would be easier to write and maintain tests if the tests for X.el were 
in X-tests.el in the same directory (or perhaps in a subdirectory 
tests/X.el), rather than in ../../test/*/*/X-tests.el, which is quite a 
bit further removed, and clumsy to navigate to (you can provide Emacs 
commands that help with this, but you really also need shell aliases, 
etc., and it's never going to be as simple as having code and tests next 
to each other).  While browsing and editing the tests, you would benefit 
from having the tests right where the source code is; and since running 
them against an older Emacs version is comparatively rare, the more 
complicated find expression that you need to collect the test files 
makes little difference.

> You're right that changing the convention is not too hard for the
> framework, but getting tests to be backward compatible is going to be
> annoying if you don't plan for it from the beginning.  In practice it
> very likely won't happen, and that's unfortunate.

I think this is a good time to discuss concrete proposals to solve this 
problem.

Christian.