Re: [PATCH] {yacc-work} tests: cover yacc target-specific flags, and `-v' flag handling

[Stefano Lattarini]

On Friday 21 January 2011, Ralf Wildenhues wrote:
> * Stefano Lattarini wrote on Fri, Jan 21, 2011 at 08:07:47PM CET:
> > On Friday 21 January 2011, Ralf Wildenhues wrote:
> > > The patch is OK, but still, can you please wait until, say, at least
> > > Monday before pushing?
> > >
> > Yes.  I just wanted to have it approved for the 'yacc-work' branch, 
> > so that I could move on with the non-testsuite work in there.
> 
> > A good idea would then be to make the 'yacc-work' branch public, without
> > merging it to master until testsuite issues settle down.  WDYT?
> 
> If that helps you, sure.
>
OK, will do then.

> > > The point of introducing new tests is lessened if we don't sometimes go
> > > around and fix the issues that running them exposes.  And we really
> > > should strive to have zero FAILures in the test suites by the next
> > > release on all but the near-museum systems.
> > >
> > Ah, but the VPATH issue with FreeBSD (which is long standing BTW) will be
> > quite tricky to fix.
>
> Which one exactly?  I'll look at it, before we declare defeat with
> ensure_distcheck_.
>
I have an half-baked bug report.  I'll try to complete and submit it soonish.

> > > Things that we cannot fix should be XFAILs instead;
> > >
> > Fact is, many failing tests fail only with some make implementations (and
> > maybe just one).  Declaring them as XFAILing whould just cause XPASSes
> > with all the other makes :-(
> 
> There are two choices: 1) we modify the test to expose the failure
> everywhere.  2) we modify the test to not have the issue anywhere.
> 
> It is very helpful if the testsuite strives to be orthogonal with
> respect to such issues: if a test is a primary exposer, then it should
> expose the issue everywhere (be that with greps or whatever else).
>
Ideally yeas, but this cannot always work reliably, especially if the
bug the test means to expose is a lack of workarounds for a limitation
or a bug in a particular make implementation.

> If it is a drive-by bug (that is not what the test was primarily set
> out to test), then we can try to hide the issue, add a FIXME in the
> test, write a bug-automake mail, and get on with our lives.
>
> Can we agree on this strategy?
>
Yes, I tried that in some previous situations (expose the bug in one
test, temporarily working around it in the others, and once the bug
gets fixed, remove those workarounds).  It tends to work out pretty
well (if not abused).

> > > distros typically won't upgrade upon test
> > > failures, so abusing that as TODO list is not a good idea in the longer
> > > term (not even short-term, as having failures for a longer time tends to
> > > blind oneself for new regressions).
> > >
> > Right -- and I'll catch this occasion to ping a patch of mine which
> > fixed what IMHO amounted to a spurious failure with Solaris make:
> >  <http://lists.gnu.org/archive/html/automake-patches/2010-11/msg00221.html>
> > (note that the patch is badly out-of-date, and needs to be rebased -- I'll
> > do that if you agree it's worth applying).
> 
> That's exactly what I mean with (2) above.  Except: I don't see the bug
> happening on Solaris: silent-yacc-generic.test passes for me even with
> ancient Solaris 2.6 make.  Of course it doesn't help that I need to set
> YACC=yacc to not have the test SKIPped in the first place.  The logic in
> tests/defs is definitely suboptimal that way.  (We should accept a
> vendor yacc if it exists; but of course we cannot expect it to accept
> --version).
>
Fixing that is in my TODO list.

> What make implementation did you test this with?  Was this on a real
> Solaris,
>
Yes, with Solaris 10 XPG4 make (I've just re-checked, JTBS).
OTOH, the test passes with Solaris CCS make.  Weird.

> or was this one of the heirloom tools again?  We really don't
> need to cater to the latter, because nobody uses them in practice.
>
Maybe, but they are very very close to their Solaris/OpenSolaris
counterparts, and very useful for rapid testing.  I agree that we
souldn't put great efforts in them (in fact, a couple of automake
tests currently causes Heirlooom make to segfault, but hey, who
cares), but if we can easily preserve a decent support for them,
we should.

> Cheers,
> Ralf
> 

Regards,
  Stefano