Re: Testing groff. Was: GNUism in groff tests

[Ralph Corderoy]

Hi Ingo,

> Werner wrote:
> > I think the proper way for testing groff would be to make it run
> > with a fuzzer,
>
> Yes, that would no doubt be useful.
...
> is kind of orthogonal to developing a test suite, though.
...
> A test suite, on the other hand, is most useful for making sure no
> regressions creep into the functionality,

I think that's groff's pressing need: stopping regressions.  Given what
I imagine is the part-time nature of contributions, to a large code
base, that implements a complex system of parts, e.g. the troff
language, I suspect a fix to more delicate bug may be quite high risk of
introducing a regression, and often this will be only apparent in the
PostScript, PDF, or pixels on the page being different; no warnings,
errors, or dumped core.

That bug I referred to earlier is a good example.  

    https://bugs.launchpad.net/ubuntu/+source/groff/+bug/42764

    The footer printed with the -mm macros has changed position between
    groff versions 1.17.2 and 1.18
    ...
        .\" groff -mm -ma4 footermoves.tr >footermoves.ps
        .\"
        .PGFORM 21c-2i 29.7c-1i 1i 1
        .PH "'hl'hc'hr'"
        .PF "'fl'fc'fr'"
        .\"
        .S +1
        .P
        Footer not in a constant place across groff versions.

    If the `.S +1' is commented out the problem doesn't occur.

Fixing known bugs, finding more bugs with fuzzing, implementing new
features, or adding something from another troff, are all less fun to do
if one's constantly concerned that there's not a good safety net to stop
faults getting released.  And a buggy groff version or two means
documents have to start testing the troff implementation and its version
to implement workarounds.

So I think regression tests come first, and they don't have to be unit
tests.  In some ways, that's the worst option because they tend to be
trivial functions being tested; it's the interplay of the whole system
that matters.  A corpus of documents and golden eyeballed output, say
ditroff, PostScript, or PNG to suit, would be a good start.  Even if
it's just a script that compares renderings of all the documents shipped
with groff between two built-but-not-installed groffs.

Do any of the other troffs have regression tests to our benefit?

-- 
Cheers, Ralph.