Re: [PATCH] {yacc-work} tests: cover yacc target-specific flags, and `-v' flag handling

[Ralf Wildenhues]

* Stefano Lattarini wrote on Fri, Jan 21, 2011 at 08:07:47PM CET:
> On Friday 21 January 2011, Ralf Wildenhues wrote:
> > The patch is OK, but still, can you please wait until, say, at least
> > Monday before pushing?
> >
> Yes.  I just wanted to have it approved for the 'yacc-work' branch, 
> so that I could move on with the non-testsuite work in there.

> A good idea would then be to make the 'yacc-work' branch public, without
> merging it to master until testsuite issues settle down.  WDYT?

If that helps you, sure.

> > The point of introducing new tests is lessened if we don't sometimes go
> > around and fix the issues that running them exposes.  And we really
> > should strive to have zero FAILures in the test suites by the next
> > release on all but the near-museum systems.
> >
> Ah, but the VPATH issue with FreeBSD (which is long standing BTW) will be
> quite tricky to fix.

Which one exactly?  I'll look at it, before we declare defeat with
ensure_distcheck_.

> > Things that we cannot fix should be XFAILs instead;
> >
> Fact is, many failing tests fail only with some make implementations (and
> maybe just one).  Declaring them as XFAILing whould just cause XPASSes
> with all the other makes :-(

There are two choices: 1) we modify the test to expose the failure
everywhere.  2) we modify the test to not have the issue anywhere.

It is very helpful if the testsuite strives to be orthogonal with
respect to such issues: if a test is a primary exposer, then it should
expose the issue everywhere (be that with greps or whatever else).
If it is a drive-by bug (that is not what the test was primarily set
out to test), then we can try to hide the issue, add a FIXME in the
test, write a bug-automake mail, and get on with our lives.

Can we agree on this strategy?

> > distros typically won't upgrade upon test
> > failures, so abusing that as TODO list is not a good idea in the longer
> > term (not even short-term, as having failures for a longer time tends to
> > blind oneself for new regressions).
> >
> Right -- and I'll catch this occasion to ping a patch of mine which
> fixed what IMHO amounted to a spurious failure with Solaris make:
>  <http://lists.gnu.org/archive/html/automake-patches/2010-11/msg00221.html>
> (note that the patch is badly out-of-date, and needs to be rebased -- I'll
> do that if you agree it's worth applying).

That's exactly what I mean with (2) above.  Except: I don't see the bug
happening on Solaris: silent-yacc-generic.test passes for me even with
ancient Solaris 2.6 make.  Of course it doesn't help that I need to set
YACC=yacc to not have the test SKIPped in the first place.  The logic in
tests/defs is definitely suboptimal that way.  (We should accept a
vendor yacc if it exists; but of course we cannot expect it to accept
--version).

What make implementation did you test this with?  Was this on a real
Solaris, or was this one of the heirloom tools again?  We really don't
need to cater to the latter, because nobody uses them in practice.

Cheers,
Ralf