sensitivity vs. specificity in software testing

[Ralph Corderoy]

Hi Branden,

> On the one hand I like the idea of detecting inadvertent changes to
> vertical spacing (or anything else) in a document, but on the other,
> I find narrowly scoped regression tests to be advantageous.

Agreed.  I assume groff is a long way from a set of tests which give
high code coverage.  I think that swings in favour of detecting
inadvertent changes.

> I think maybe the best-of-both-worlds solution is to have a model
> document-based automated test--perhaps one that exercises as many
> ms(7) macros as possible.

A bit of a torture test?  Yes, worthy.

> would add the highly sensitive Rumsfeldian "unknown unknowns" problem
> detection that I think your suggestion is tuned to.

I don't think it would catch the things not thought of, like the .bp
within a display.  I've probably mentioned it before, but a corpus of
real-life documents would be good input to a troff test harness.  Render
each at, say, 150 pixels per inch in monochrome by default and compare
against a golden version made earlier.

Commands like ‘gm compare’ or gmic(1) can do the pixel comparison.
A differing pixel would leave the diffs of each stage of the pipeline
for eyeballing.  So tbl's output is saved rather than just piped into
troff.

This needn't be part of groff's test suite, or anything to do with the
FSF.  It could be useful for comparing versions of other troffs.
Documents could be tagged with what troffs they're compatible with and
whether the test needs upgrading, say to colour pixels.  A buildbot or
similar could run the suite against a new release or Git commit.

When a new document is thrown into the pot, a golden test result is
eyeballed and saved.  It's not too important whether it's perfect so it
doesn't need costly proof-reading.  What matters is a later
change is detected and ensured to be deliberate.

> >     output=\
> >     ',,,,,,The first page is 1.,,     display,,,,,,,,,
> >     ,,,                             -2-,,,The second page is 2.
> >     '
> >     output=$(echo "$output" | tr , \\012)
>
> This is a good suggestion for handling blank line-happy output, of
> which we have quite a bit in groff.

I of course produced it by doing the opposite, line feeds into commas,
and then reverted a comma by hand where I wanted to show a page break
with a linefeed.

-- 
Cheers, Ralph.