Re: [Orgmode] How you can help

[Tom Breton (Tehom)]

> "Avdi Grimm" <address@hidden> writes:
> > A few points, from someone with a decent amount of testing background:
>
> Jippie! Please stay with us for a few days :-D

[...]

As someone with also a decent amount of testing background, I'd like
to answer also.

> Did you work with unit-testing frameworks for elisp already?
> Which one?

I co-wrote regress.el (Wayne Mesard wrote the first version) and I
wrote rtest.  I've written some testing for allout.el before I
switched over to org-mode (which is very nice, BTW, Carsten, and
provides pretty much everything I wish-listed for allout mode).  Of
particular interest for testing org-mode, I wrote mockbuf.el, which mocks
files and buffers so that tests can deal with files without changing
anything permanently.

> Recommendations?

Hard for me to say objectively.

I prefer my rtest but it's in flux right now.  I wouldn't recommend
the current public version, rtest 3.0  I've designed and half written a
better version but I can't really ask you to wait until I get around
to making a better version of rtest.  There is a development version
that works and does nice things; it's messy as hell right now and I'm
embarrassed about that.  It isn't public but I could put it up if
you'll forgive its messy state,

It has mockbuf, which I found indispensable in writing tests for
allout.el.  In fact, I mostly developed it for that, though all the
allout-specific code is elsewhere.  Also useful for writing tests:
safe-equal.el, caselet.el, and select.cl, all in the ancillary
directory.

I would steer clear of frameworks that provide many of synonyms for
equality tests, inequality tests, etc.  You already know how to write
(not (equal ...)) in elisp.  Why learn the name and syntax of
something like (shouldnt-be-equal A B)?  Having a lot of that makes a
package LOOK heftier but doesn't do anything more for you.  IMO
analyzing test failures should be done by inspecting the test form,
not by making you write things in non-standard elisp that could be
written in standard elisp.

Using ert would mean requiring ewoc, which means tests wouldn't work
on emacs 21.  Too bad, because it has some appeal.



> Could say something about the effort to get started with such a
> framework?

For rtest, right now you just need (a) the development version, (b)
unzip it in your emacs-site-lisp or set the paths and autoloads
manually if you prefer, (c) include the following in a file to be
tested:

        (when (not (fboundp 'rtest:deftest))
            (defmacro rtest:deftest (&rest dummy))
            (defmacro rtest:if-avail (&rest dummy)))

Then write tests as:

        (rtest:deftest function-name
           ("Docstring"
              (test-form) ;;Return a boolean
              )
           ;;Repeat as wanted
           )

That will get you started, but really you'd want to get familiar with
the stuff in mockbuf.el - which is so messy right now that I'm
embarrassed about it, because it contains both old and new versions of
many things in flux.

Other advice: Make a directory to hold short examples of files that
illustrate "before" and "after" states for the various
transformations.  Each unit (eg org-agenda.el, org-archive.el etc)
should have its own subtree of examples.

One thing I'd do early is build the following test helper functions:

A macro that runs a test body inside a "let" of all the configuration
options set to expected values.

A function to compare vector-style strings.  `equal' doesn't suffice.
rtest has, in ancillary/safe-equal.el, safe-equal-compare-objects and
safe-equal-compare-vectors, which IIRC suffice for this.

The ability to write only the visible parts to a separate buffer was
indispensable in testing allout's folding.  You already have it in
org-export-visible with the type that keeps the buffer, "?\ ".  Then
you can just compare contents.

You will likely have a lot of what I call "output tests", tests of
code that has the property that if the output looks right, it is
right.

IMO with output tests, you need to make it easy on yourself.  TDD is
all well and fine, but not if it makes you repeat yourself.  If there
is available output that looks right, capture it and use it as a base
for comparison.  That's not "pure" but it's far easier and it works
right.  If the available output isn't yet right, capture it anyways
and tweak it to make it right - but this rarely happens.

I automated that sort of thing for a Prolog test harness I wrote,
works fine.  You'd just write a test of a special type that produced
and captured output.  It would originally be ungraded, but you would
then grade it, and it would remember the correct output and compare
against it.

I was planning to add the same to rtest and I partly did, but never
finished it.  Still on my todo list.

> Can we add it without changing Org's code? If understand 2. correctly -
yes?
>

rtest's tests like to be in the same file as the code.  If you use the
form above, the file still compiles and runs even when rtest is not
available.

It's possible to write the tests in another file, but then either
tests aren't grouped together (must be run one by one) or you have to
do extra work to group them.  If they're in file, you can run every
test in a package with `rtest:library'.  The other really convenient
way to run a test is rtest:defun-at-point.


> If we can, would we loose quality/speed of tests?

Not at all.


Tom Breton (Tehom)

PS: One more piece of advice: Make an auto-insert or at least a
skeleton that inserts test headers, like the "(when (not (fboundp ..."
stuff above.  You'll be using it a fair bit, and it's real nice to
have it just automatically be there when you start a new file.  This
advice is geared toward my way of working, though, with the test code
in the same file as the functional code.