[bug #62478] add more documentation of test suite usage

[G. Branden Robinson]

Update of bug #62478 (project groff):

                Severity:              3 - Normal => 1 - Wish               
              Item Group:                    Test => Documentation          
                  Status:                    None => In Progress            
             Assigned to:                    None => gbranden               
                 Summary: Make test files testable everywhere => add more
documentation of test suite usage

    _______________________________________________________

Follow-up Comment #2:

Hi Bjarni,

I don't agree with much of the premise of this bug.  There is no reason for
the test scripts to be "runnable from anywhere".  They need to be runnable
straightforwardly from the build tree, because that is where they are intended
to be used.

I have asserted many times in discussions with you about the groff test suite
that it is not intended to be a test suite for *roff systems in general.  It
is not even intended as a test suite for an existing groff installation.  Its
purpose is to check the behavior of the groff system _that has just been
compiled on a host_ for correctness.

Anything else is out of scope.  Development resources are limited enough
without expanding that scope.

If someone wants to turn the 119 test scripts into some other kind of tool,
they are welcome to do so under the terms of the GPL.

Nevertheless, I am expanding the "In Case of Trouble" section of the
"INSTALL.extra" document.  When I push my next batch of commits I expect it to
look much like this.


In Case of Trouble
==================

If a test fails, gather its log file from the build directory.  For
instance, the test "tmac/tests/localization-works.sh" (in the source
directory) will have a log file called
"tmac/tests/localization-works.sh.log" in the build directory.

To re-run a test, change to the top of the build directory (if
necessary) and run the test by name from the shell prompt.

For example, to rerun the test mentioned above from a "build" directory
I created as a subdirectory in the source tree, I would do this.

  (cd build && ../tmac/tests/localization-works.sh)

I can view the test log as follows.

  cat build/tmac/tests/localization-works.sh.log

Many known issues are documented in the PROBLEMS file; some of them are
historical.  You can browse groff bug reports via the GNU Savannah issue
tracker.

  https://savannah.gnu.org/bugs/?group=groff

If that doesn't help and you need support, please contact the groff
mailing list at groff@gnu.org.  If you think that you have found a bug,
please submit a ticket.

  https://savannah.gnu.org/bugs/?group=groff&func=additem


You said:
> Tests, that produce FAIL or SKIP, should be fit to be analysed by the users
anywhere they like, the simplest case in the same directory as the file
resides.

This is already the case.  They need only look at the log file, as documented
above.

> In the output with FAIL or SKIP status, the users should get instructions to
define an environment[] variable "abs_top_builddir" to be the path to their
groff build directory ...

This simply isn't necessary.  They can re-run the test from the build
directory itself.  That is the only supported scenario.  Anything else adds
variables (in the experimental sense), potentially confounding testing.

> There is already a file,
src/devices/grotty/tests/basic_latin_glyphs_map_correctly.sh, that can profit
from this.

It might profit from the attachment of the log file, which I need to ask John
Gardner for.  Assuming the log file has useful output, that is--it might not,
and if that is the case the problem might be on my own head.  But this aside
is best carried on in bug #62357.

> No test file should produce the XFAIL or SKIP case except
> a) the groff code still has a bug (excluding the test file)

Yes, that's roughly my understanding of the denotation of XFAIL.

> b) the complete operation system does not provide the absolute necessary
resources

This statement gestures in the direction of the meaning of SKIP, but is too
narrowly worded.  Tests could be skipped if the portion of the system under
test were subject to disablement at configuration time.  For example, I think
X11 support is optional, though I've never tried to build without it.  If
gxditview had tests, which it does not, those would reasonably be SKIPped in a
scenario that deliberately didn't build the program.

I hope my proposed documentation change addresses your concerns, modulo the
philosophical differences.


    _______________________________________________________

Reply to this item at:

  <https://savannah.gnu.org/bugs/?62478>

_______________________________________________
  Message sent via Savannah
  https://savannah.gnu.org/