Re: testing framework and package.el

[Sebastian Rose]

Christian Ohler <address@hidden> writes:
>  On 2/10/10 13:53, Sebastian Rose wrote:
>> Below that directory, a tree like the source tree exists:
>>
>> project
>>     +-- lisp/
>>     |     +-- a.el
>>     |     `-- b/
>>     |         +-- b.el
>>     |
>>     `-- tests/
>>           +-- a.el/
>>           |     +-- tests.el
>>           |     `-- a-defun.el
>>           `-- b/
>>               +-- b.el/
>>                     +-- tests.el
>>                     `-- b-defun.el
>>
>> If this setup exists, when editing defun-x in lisp/a.el,
>> `M-x org-test-test-current-defun' will load tests/a.el/defun-x.el
>> (fallback: tests.el there) and execute all tests with selector
>> "^a-defun".
>> `M-x org-test-test-buffer-file' in that same source file will load all
>> *.el files in tests/a.el/ and execute all ERT tests for selector "^a".
>>
>> This would   a) recommend a certain structure for testing (what ever the
>> structure will be) and   b) make testing a key-stroke.
> This structure looks like it is optimized for many test files for each file of
> non-test code.  Right now, Emacs has 1735 .el files, and 35 files total in the
> tests/ directory.  This ratio is not likely to invert any time soon.  The
> structure that you describe could work, but isn't it overkill?


Might be overkill, yes.  All I wrote here was just after the first
glance at ERT and few experiments.  You're not the only one who says so
:)
I'll drop that probably.  A structure like that could be enforced later
if needed.  We will start with a very small project.  I want to make it
easy to jump to the test-file for a given source-file and execute tests
for current defun automatically.  This could be done based on such a
structure (i.e. conventions).

I guess reading more of ERT's code I'll find other ways to find the file
a certain test (or tests matching a selector) is defined.


>> 2.)  Create temp buffers for output
>>
>> One thing I miss is a way to create output buffers automagically (I
>> didn't look deeper into the sources yet).
>>
>> I know Lennart Borgman has written something for nXhtml.  It's called
>> ert2.el and part of the nXhtml distribution (but I did not comprehend
>> what's going on there yet - my elisp is badbadbad :).  Maybe Lennart
>> will help here or even care for that addition?
> Lennart's code is already integrated in ERT, obsoleting ert2.  I remember 
> that I
> wasn't quite happy with his design, but didn't have time to fix it, and since 
> it
> solved an important problem, I included it as an experimental addition in
> ert-exp.el.

OK, good.  Found that, thanks.  From skimming over that file I guess
that's what I was looking for and maybe even a bit more.


>> 3.)  Compare output to control files
>>
>> ediff the contents of those output buffers with control files.  We need
>> this feature to test org's exports.  No one will find the time to sit
>> there and search hundreds of output files for errors "by hand" (once
>> tests exist).
> What do you mean by "sitting there and searching hundreds of output files for
> errors"?  Do you have an example of tests where ERT's error reporting is too
> unspecific to be useful?
>
> I remember writing some tests roughly like this:
>
> (ert-deftest foo ()
>   (with-temp-buffer
>     (insert "initial buffer contents")
>     (... invoke code under test ...)
>     (should (equal (ert-filter-string (buffer-string) ...)
>                    "expected final buffer contents"))))
>
> Are you sure that you want the initial/expected buffer contents in their own
> files?  Such a split between the inputs and the invocation of the code under
> test seems cumbersome.  I guess if your inputs/expected outputs are complex
> enough, moving them out into files can make sense.  Can you point me to some
> examples where you think the data is too complex to be in-line?


Org's exports to XHTML, LaTeX etc.  We could compare them to control
files.  The publishing mechanism might create quite big files and even
entire file/directory trees.  Currently we publish our stuff and detect
regressions when clicking a link that points nowhere or validating a
XHTML page (spot checking).



> One limitations of my approach is that we really need some kind of
> "with-test-buffer" rather than "with-temp-buffer" that keeps the buffer around
> for inspection if the test fails, but this is item 2.) on your list already.
> Another limitation is that the explainer for `equal' is not as pretty as 
> ediff,
> but OTOH, as soon as you want to compare text properties, ediff can't help
> anymore (AFAIK), while there is an explainer for
> ert-equal-including-properties'.


Also, I believe ediff is too interactive for a simple test run (diff's
exit status is 0 if no difference is found - don't know yet if and to
how ediff could be used non-interactively).


> Text properties (e.g., font lock properties) are important to many kinds of
> tests.  Both ediff and using files are problematic in such cases because they
> don't support text properties.  Do text properties never come up in your
> context?


Some so some so I guess.  I didn't start to write actual tests yet.
This will be done by many people (I hope) and there will be many
different types of tests (I hope).  Some will involve text properties,
too.



Thank's for your comments!  I think I have enough to fill my spare time
this week :)


  Sebastian