Re: tests updates

[Stefano Lattarini]

On Wednesday 03 November 2010, Ralf Wildenhues wrote:
> * Stefano Lattarini wrote on Wed, Nov 03, 2010 at 02:27:47PM CET:
> > On Monday 01 November 2010, Ralf Wildenhues wrote:
> > > I'm so totally behind on patches and not getting better, that the
> > > strategy of ignoring testsuite work will not help either.  So how about
> > > the following.  IIRC you suggested a branch for low-danger testsuite
> > > updates.  I'm not sure if a single branch would always be the right
> > > thing to do, often things that are independent can better be treated
> > > in independent branches.  But anyway, how about if you go ahead with
> > > the idea of merging such patches, preferably based off of maint and
> > > merge to master (unless there are really good reasons why branch-1.11
> > > would need them too), with the following strategy similar to Libtool:
> > > you post the patches, and push them after 72 hours if no feedback by
> > > then.  WDYT?  Does that sound good enough for you?
> > Definitely yes, thanks!
> 
> Good.  I wouldn't want to lose your work just over my lack of time.
> 
> > Just one question: what about the already-existing "tests-init" branch?
> > Should I try to bring it to a point where it can be easily merged into
> > master, and then forget about it, or should it continue to live parallel
> > to master?
> 
> Right now the branch looks like its changes would be fairly safe to
> merge to master, but also, it would mean that sharing testsuite patches
> between branch-1.11 and master may become a bit more cumbersome.  So I'd
> say it depends a bit on where you want to go with this in the long run.
Well, I'd like to:

 1. reorganize the code in "tests/defs.in" in a more rational way
    (basically just a reordering of the code);

 2. (re)introduce the `run_command' function and use it throughout;

 3. make it easier for the user to override the aclocal "search path"
    in the testsuite (a new pending patch from Paolo Bonzini could
    help with this);

 4. improve requirements' declaration in Automake test scripts, and
    make it easier to run the Automake testsuite using different C,
    C++ and Fortran compilers, without leaning too much towards the
    GNU compilers as it does now.

These major changes would be interspesed with minor code cleanups
and refactoring, such as (to list a few) avoiding to leak temporary
variables from `tests/defs' into the test scripts, renaming `$curdir'
to (say) `$testbuilddir' for better clarity, and improve the messages
from skipped tests.

> We can easily move to a strategy of having new testsuite stuff added to
> master only by default, and only care about branch-1.11 for fixing
> existing regressions and serious bugs, which would probably make the
> situation easier.
Yes, I'd like to go this way.
 
> Of course there are other active branches too, that shouldn't have a
> harder-than-necessary time.
Ideally, all the tests that work with current `tests/defs' should
continue to work with the new one; the only problem might be that
some new tests committed to another branch might cause *maintcheck*
failures when merged to master, but that is not a big deal IMHO,
and easily fixed anyway.
 
> We could treat merges similarly to patches in announcing the intention
> to merge 72h before the fact, in those cases where there is doubt over
> the merge (or the precise point at which a merge should be done).
This won't be necessary if we stick to master for tests extensions and
refactoring.
> I must admit that I don't have all that much experience with merges of
> longer-term branches done by more than one person; one alternative would
> be that you suggest the merge and I do it (if that works out).  Testing
> such merges usually requires a bit of additional work, in that both
> branches need to be checked for global changes that need adjusting in
> the respective other branch.  Let's see how this works out.

In the end, are you OK with having me to merge "test-init" to master right
away, and do future testsuite work on master only?

Thanks,
   Stefano