Re: Parallel test execution: new option `parallel-tests': [1/4]

[Akim Demaille]

>>> "RW" == Ralf Wildenhues <address@hidden> writes:

 > +    case fail=$$fail:xpass=$$xpass:xfail=$$xfail in                 \
 > +      fail=0:xpass=0:xfail=0)                                       \
 > +        msg="$$All$$all $$tests passed.  ";                         \
 > +        exit=true;;                                                 \
 > +      fail=0:xpass=0:xfail=*)                                       \
 > +        msg="$$All$$all $$tests behaved as expected";               \
 > +        if test "$$xfail" -eq 1; then xfailures=failure;            \
 > +        else xfailures=failures; fi;                                \
 > +        msg="$$msg ($$xfail expected $$xfailures).  ";              \
 > +        exit=true;;                                                 \
 > +      fail=*:xpass=0:xfail=*)                                       \
 > +        msg="$$fail of $$all $$tests failed.  ";                    \
 > +        exit=false;;                                                \
 > +      fail=*:xpass=*:xfail=*)                                       \
 > +        msg="$$failures of $$all $$tests did not behave as expected"; \
 > +        if test "$$xpass" -eq 1; then xpasses=pass;                 \
 > +        else xpasses=passes; fi;                                    \
 > +        msg="$$msg ($$xpass unexpected $$xpasses).  ";              \
 > +        exit=false;;                                                \
 > +      *)                                                            \
 > +        echo >&2 "incorrect case"; exit 4;;                         \
 > +    esac;                                                           \
 > +    if test "$$skip" -ne 0; then                                    \
 > +      if test "$$skip" -eq 1; then                                  \
 > +        msg="$$msg($$skip test was not run).  ";                    \
 > +      else                                                          \
 > +        msg="$$msg($$skip tests were not run).  ";                  \
 > +      fi;                                                           \
 > +    fi;                                                             \

The logic to create this kind of message is really complex, and in
retrospect, I don't think that the message delivered is much easier to
read than something more regular.  So currently check.mk does as
follows:

        case fail=$$fail:xpass=$$xpass in                               \
          fail=0:xpass=0)                                               \
            msg="The test suite passed.  ";                             \
            exit=true;;                                                 \
          *)                                                            \
            msg="The test suite failed.  ";                             \
            exit=false;;                                                \
        esac;                                                           \
        msg="$$msg   - $$pass/$$all tests passed.  ";                   \
        test $$fail -eq 0 ||                                            \
          msg="$$msg   - $$fail failures.  ";                           \
        test $$xpass -eq 0 ||                                           \
          msg="$$msg   - $$xpass unexpected pass.  ";                   \
        test $$xfail -eq 0 ||                                           \
          msg="$$msg   - $$xfail expected failures.  ";                 \
        test $$tfail -eq 0 ||                                           \
          msg="$$msg   - $$tfail expected temporary failures.  ";       \
        test $$skip -eq 0 ||                                            \
          msg="$$msg   - $$skip skipped tests.  ";                      \


It's simpler to create the message, and I think, the message is also
simpler.


We introduced TFAIL, temporary failures: it's sort of short-term
XFAIL.  When a test starts to fail, it might be because someone
uncovered a bug elsewhere, and that person might not be competent to
address the real bug.  Yet, leaving the test as is, failing, is a
problem for everybody else who might waste a lot of time to discover
that they have not introduced the problem, it was already there.

So we make it TFAIL, which means "should be processed soon".