Parallel automake execution: AUTOMAKE_JOBS. [1/4]

[Ralf Wildenhues]

This patch provides the basic functionality.  It aims to provide
unchanged code paths for the non-threaded case.

Notable hack in Channels.pm: before exiting due to an internal error,
it is necessary to flush stderr.  Otherwise, if a worker thread exits
here, its output may not appear.  Strictly speaking, it is bad style
anyway to exit the program from a non-master thread, but since it
happens only in situations where we are borked anyway, I'm not so
worried.  Might be fixed later.

I've put Joakim in THANKS.

Cheers,
Ralf

        Parallel automake execution: AUTOMAKE_JOBS.

        * lib/Automake/Config.in (perl_threads): New global.
        * automake.in: Use it.  If the perl supports interpreter-based
        threading, then use `threads' and `Thread::Queue'.
        (handle_makefile, handle_makefiles_serial): New functions,
        factored out from main.
        (get_number_of_threads): New function, compute number of threads
        to use, based on environment variable `AUTOMAKE_JOBS' and number
        of independent makefiles.
        (handle_makefiles_threaded): New function.  Spawn threads, use
        thread queue to distribute handling the different makefiles.
        Collect $exit_code values from threads.
        (main): Use new functions.
        * aclocal.in: No threads here.
        * configure.ac: Substitute PERL_THREADS; enabled with perl >=
        5.7.2 and when ithreads are available.
        * bootstrap (dosubst): Likewise.
        * Makefile.am (do_subst): Likewise.
        * lib/Automake/Makefile.am (do_subst): Likewise.
        * lib/Automake/ChannelDefs.pm: Use `Automake::Config' and
        `threads'.
        (verb): Prepend thread ID (tid) to verbose messages.
        * lib/Automake/Channels.pm (msg): Before exiting, flush stderr,
        needed for worker threads.
        * lib/Automake/tests/Makefile.am (TESTS_ENVIRONMENT): Also
        include the build tree path, so Config.pm is found.
        * tests/parallel-am.test: New test.
        * tests/Makefile.am: Update.
        * doc/automake.texi (Invoking Automake): Document AUTOMAKE_JOBS.
        * NEWS, THANKS: Update.
        Report about long execution times by Joakim Tjernlund and others.

diff --git a/Makefile.am b/Makefile.am
index 3924daa..8b83cad 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -70,6 +70,7 @@ do_subst = sed \
   -e 's,address@hidden@],$(PACKAGE),g' \
   -e 's,address@hidden@],$(PATH_SEPARATOR),g' \
   -e 's,address@hidden@],$(PERL),g' \
+  -e 's,address@hidden@],$(PERL_THREADS),g' \
   -e 's,address@hidden@],$(SHELL),g' \
   -e 's,address@hidden@],$(VERSION),g' \
   -e 's,address@hidden@],Generated from address@hidden; do not edit by 
hand.,g' \
diff --git a/NEWS b/NEWS
index dce4bbd..9d123f1 100644
--- a/NEWS
+++ b/NEWS
@@ -10,7 +10,15 @@ New in 1.10a:
     (and new in Automake 1.10) is degraded to a warning.  This helps
     in the common case where the Autoconf versions used are compatible.
 
-* Change to Libtool support:
+* Changes to automake:
+
+  - The automake program can run multiple threads for creating most
+    Makefile.in files concurrently, if at least Perl 5.7.2 is available
+    with interpreter-based threads enabled.  Set the environment variable
+    AUTOMAKE_JOBS to the maximum number of threads to use, in order to
+    enable this experimental feature.
+
+* Changes to Libtool support:
 
   - Libtool generic flags are now passed to the install and uninstall
     modes as well.
diff --git a/aclocal.in b/aclocal.in
index 44d3d1e..3cfe1c0 100644
--- a/aclocal.in
+++ b/aclocal.in
@@ -47,6 +47,9 @@ use Cwd;
 
 # Some globals.
 
+# We do not operate in threaded mode.
+$perl_threads = 0;
+
 # Include paths for searching macros.  We search macros in this order:
 # user-supplied directories first, then the directory containing the
 # automake macros, and finally the system-wide directories for
diff --git a/automake.in b/automake.in
index fabeb48..c9b1c63 100755
--- a/automake.in
+++ b/automake.in
@@ -128,6 +128,11 @@ package Automake;
 
 use strict;
 use Automake::Config;
+if ($perl_threads)
+  {
+    use threads;
+    use Thread::Queue;
+  }
 use Automake::General;
 use Automake::XFile;
 use Automake::Channels;
@@ -7984,6 +7989,123 @@ sub parse_arguments ()
     if $errspec && ! @input_files;
 }
 
+
+# handle_makefile ($MAKEFILE_IN)
+# ------------------------------
+# Deal with $MAKEFILE_IN.
+sub handle_makefile ($)
+{
+  my ($file) =  @_;
+  ($am_file = $file) =~ s/\.in$//;
+  if (! -f ($am_file . '.am'))
+    {
+      error "`$am_file.am' does not exist";
+    }
+  else
+    {
+      # Any warning setting now local to this Makefile.am.
+      dup_channel_setup;
+
+      generate_makefile ($am_file . '.am', $file);
+
+      # Back out any warning setting.
+      drop_channel_setup;
+    }
+}
+
+# handle_makefiles_serial ()
+# --------------------------
+# Deal with all makefiles, without threads.
+sub handle_makefiles_serial ()
+{
+  foreach my $file (@input_files)
+    {
+      handle_makefile ($file);
+    }
+}
+
+# get_number_of_threads ()
+# ------------------------
+# Logic for deciding how many worker threads to use.
+sub get_number_of_threads
+{
+  my $nthreads = $ENV{'AUTOMAKE_JOBS'} || 0;
+
+  $nthreads = 0
+    unless $nthreads =~ /^[0-9]+$/;
+
+  # It doesn't make sense to use more threads than makefiles,
+  my $max_threads = @input_files;
+
+  # but a single worker thread is helpful for exposing bugs.
+  if ($automake_will_process_aux_dir && $max_threads > 1)
+    {
+      $max_threads--;
+    }
+  if ($nthreads > $max_threads)
+    {
+      $nthreads = $max_threads;
+    }
+
+  # We cannot deal with --add-missing (yet).
+  $nthreads = 0
+    if ($add_missing);
+
+  return $nthreads;
+}
+
+# handle_makefiles_threaded ($NTHREADS)
+# -------------------------------------
+# Deal with all makefiles, using threads.  The general strategy is to
+# spawn NTHREADS worker threads, and dispatch makefiles to them.
+sub handle_makefiles_threaded ($)
+{
+  my ($nthreads) = @_;
+
+  my @queued_input_files = @input_files;
+  my $last_input_file = undef;
+  if ($automake_will_process_aux_dir)
+    {
+      $last_input_file = pop @queued_input_files;
+    }
+
+  # The file queue distributes all makefiles.
+  my $file_queue = Thread::Queue->new;
+
+  verb "spawning $nthreads worker threads";
+  my @threads = (1 .. $nthreads);
+  foreach my $t (@threads)
+    {
+      $t = threads->new (sub
+       {
+         while (my $file = $file_queue->dequeue)
+           {
+             verb "handling $file";
+             handle_makefile ($file);
+           }
+         return $exit_code;
+       });
+    }
+
+  # Queue all normal makefiles.
+  verb "queuing " . @queued_input_files . " input files";
+  $file_queue->enqueue (@queued_input_files, (undef) x @threads);
+
+  foreach my $t (@threads)
+    {
+      my @exit_thread = $t->join;
+      $exit_code = $exit_thread[0]
+       if ($exit_thread[0] > $exit_code);
+    }
+
+  # The master processes the last file.
+  if ($automake_will_process_aux_dir)
+    {
+      verb "processing last input file";
+      handle_makefile ($last_input_file);
+    }
+}
+
 ################################################################
 
 # Parse the WARNINGS environment variable.
@@ -8005,24 +8127,15 @@ if (! @input_files)
     fatal ("no `Makefile.am' found for any configure output$msg");
   }
 
-# Now do all the work on each file.
-foreach my $file (@input_files)
-  {
-    ($am_file = $file) =~ s/\.in$//;
-    if (! -f ($am_file . '.am'))
-      {
-       error "`$am_file.am' does not exist";
-      }
-    else
-      {
-       # Any warning setting now local to this Makefile.am.
-       dup_channel_setup;
-
-       generate_makefile ($am_file . '.am', $file);
+my $nthreads = get_number_of_threads ();
 
-       # Back out any warning setting.
-       drop_channel_setup;
-      }
+if ($perl_threads && $nthreads >= 1)
+  {
+    handle_makefiles_threaded ($nthreads);
+  }
+else
+  {
+    handle_makefiles_serial ();
   }
 
 exit $exit_code;
diff --git a/bootstrap b/bootstrap
index aa50f5d..8437d11 100755
--- a/bootstrap
+++ b/bootstrap
@@ -56,6 +56,7 @@ fi
 VERSION=`sed -ne '/AC_INIT/s/^[^[]*\[[^[]*\[\([^]]*\)\].*$/\1/p' configure.ac`
 PACKAGE=automake
 datadir=.
+PERL_THREADS=0
 
 # Override SHELL.  This is required on DJGPP so that Perl's system()
 # uses bash, not COMMAND.COM which doesn't quote arguments properly.
@@ -100,6 +101,7 @@ dosubst ()
   sed -e "address@hidden@%$APIVERSION%g" \
       -e "address@hidden@%$PACKAGE%g" \
       -e "address@hidden@%$PERL%g" \
+      -e "address@hidden@%$PERL_THREADS%g" \
       -e "address@hidden@%$BOOTSTRAP_SHELL%g" \
       -e "address@hidden@%$VERSION%g" \
       -e "address@hidden@%$datadir%g" \
diff --git a/configure.ac b/configure.ac
index a21bfb2..2bec317 100644
--- a/configure.ac
+++ b/configure.ac
@@ -60,6 +60,30 @@ installed, select the one Automake should use using
   ./configure PERL=/path/to/perl])
 }
 
+# We require ithreads support, and version 5.7.2 for CLONE.
+AC_CACHE_CHECK([whether $PERL supports ithreads], [am_cv_prog_PERL_ithreads],
+[if $PERL -e '
+    require 5.007_002;
+    use Config;
+    if ($Config{useithreads})
+      {
+       use threads;
+       use Thread::Queue;
+       exit 0;
+      }
+    exit 1;' >&AS_MESSAGE_LOG_FD 2>&1
+then
+  am_cv_prog_PERL_ithreads=yes
+else
+  am_cv_prog_PERL_ithreads=no
+fi])
+if test $am_cv_prog_PERL_ithreads = yes; then
+  PERL_THREADS=1;
+else
+  PERL_THREADS=0;
+fi
+AC_SUBST([PERL_THREADS])
+
 # The test suite will skip some tests if tex is absent.
 AC_CHECK_PROG([TEX], [tex], [tex])
 
diff --git a/doc/automake.texi b/doc/automake.texi
index 122b977..cd2f920 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -2569,6 +2569,12 @@ for this reason.
 
 @end table
 
address@hidden AUTOMAKE_JOBS
+If the environment variable @env{AUTOMAKE_JOBS} contains a positive
+number, it is taken as the maximum number of Perl threads to use in
address@hidden for generating multiple @file{Makefile.in} files
+concurrently.  This is an experimental feature.
+
 
 @node configure
 @chapter Scanning @file{configure.ac}
diff --git a/lib/Automake/ChannelDefs.pm b/lib/Automake/ChannelDefs.pm
index eff5d52..17038a6 100644
--- a/lib/Automake/ChannelDefs.pm
+++ b/lib/Automake/ChannelDefs.pm
@@ -15,6 +15,11 @@
 
 package Automake::ChannelDefs;
 
+use Automake::Config;
+if ($perl_threads)
+  {
+    use threads;
+  }
 use Automake::Channels;
 
 =head1 NAME
@@ -227,6 +232,8 @@ C<--verbose> messages.
 sub verb ($;%)
 {
   my ($msg, %opts) = @_;
+  $msg = "thread " . threads->tid . ": " . $msg
+    if $perl_threads;
   msg 'verb', '', $msg, %opts;
 }
 
diff --git a/lib/Automake/Channels.pm b/lib/Automake/Channels.pm
index c8a216a..d12bb8d 100644
--- a/lib/Automake/Channels.pm
+++ b/lib/Automake/Channels.pm
@@ -1,4 +1,4 @@
-# Copyright (C) 2002, 2004, 2006 Free Software Foundation, Inc.
+# Copyright (C) 2002, 2004, 2006, 2008 Free Software Foundation, Inc.
 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -566,7 +566,12 @@ sub msg ($$;$%)
 
       # Die on fatal messages.
       confess if $opts{'backtrace'};
-      exit $exit_code if $opts{'type'} eq 'fatal';
+      if ($opts{'type'} eq 'fatal')
+        {
+         # flush messages explicitly here, needed in worker threads.
+         STDERR->flush;
+         exit $exit_code;
+       }
     }
 }
 
diff --git a/lib/Automake/Config.in b/lib/Automake/Config.in
index 94caf3e..9c24a57 100644
--- a/lib/Automake/Config.in
+++ b/lib/Automake/Config.in
@@ -1,4 +1,4 @@
-# Copyright (C) 2003, 2004  Free Software Foundation, Inc.      -*- Perl -*-
+# Copyright (C) 2003, 2004, 2008  Free Software Foundation, Inc.  -*- Perl -*-
 # @configure_input@
 
 # This program is free software; you can redistribute it and/or modify
@@ -21,7 +21,7 @@ use 5.006;
 require Exporter;
 
 our @ISA = qw (Exporter);
-our @EXPORT = qw ($APIVERSION $PACKAGE $VERSION $libdir);
+our @EXPORT = qw ($APIVERSION $PACKAGE $VERSION $libdir $perl_threads);
 
 # Parameters set by configure.  Not to be changed.  NOTE: assign
 # VERSION as string so that e.g. version 0.30 will print correctly.
@@ -29,6 +29,7 @@ our $APIVERSION = '@APIVERSION@';
 our $PACKAGE = '@PACKAGE@';
 our $VERSION = '@VERSION@';
 our $libdir = '@datadir@/@address@hidden@APIVERSION@';
+our $perl_threads = @PERL_THREADS@;
 
 1;;
 
diff --git a/lib/Automake/Makefile.am b/lib/Automake/Makefile.am
index e68c901..ca5471c 100644
--- a/lib/Automake/Makefile.am
+++ b/lib/Automake/Makefile.am
@@ -1,6 +1,6 @@
 ## Process this file with automake to create Makefile.in
 
-## Copyright (C) 2001, 2002, 2003, 2004  Free Software Foundation, Inc.
+## Copyright (C) 2001, 2002, 2003, 2004, 2008  Free Software Foundation, Inc.
 
 ## This program is free software; you can redistribute it and/or modify
 ## it under the terms of the GNU General Public License as published by
@@ -51,6 +51,7 @@ do_subst = in=`echo $@ | sed 's/\.[^.]*$$//'`; sed \
   -e 's,address@hidden@],$(APIVERSION),g' \
   -e 's,address@hidden@],$(PACKAGE),g' \
   -e 's,address@hidden@],$(PERL),g' \
+  -e 's,address@hidden@],$(PERL_THREADS),g' \
   -e 's,address@hidden@],$(SHELL),g' \
   -e 's,address@hidden@],$(VERSION),g' \
   -e "s,address@hidden@],Generated from $$in.in; do not edit by hand.,g" \
diff --git a/lib/Automake/tests/Makefile.am b/lib/Automake/tests/Makefile.am
index 529a02f..acd769f 100644
--- a/lib/Automake/tests/Makefile.am
+++ b/lib/Automake/tests/Makefile.am
@@ -15,7 +15,7 @@
 ## You should have received a copy of the GNU General Public License
 ## along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-TESTS_ENVIRONMENT = $(PERL) -Mstrict -I $(top_srcdir)/lib -w
+TESTS_ENVIRONMENT = $(PERL) -Mstrict -I ../.. -I $(top_srcdir)/lib -w
 TESTS = \
 Condition.pl \
 Condition-t.pl \
diff --git a/tests/Makefile.am b/tests/Makefile.am
index 77dd1f1..483849c 100644
--- a/tests/Makefile.am
+++ b/tests/Makefile.am
@@ -449,6 +449,7 @@ output12.test \
 output13.test \
 output-order.test \
 overrid.test \
+parallel-am.test \
 parse.test \
 percent.test \
 percent2.test \
diff --git a/tests/parallel-am.test b/tests/parallel-am.test
new file mode 100755
index 0000000..f6e4743
--- /dev/null
+++ b/tests/parallel-am.test
@@ -0,0 +1,92 @@
+#! /bin/sh
+# Copyright (C) 2008  Free Software Foundation, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+# Test parallel automake execution.
+
+# There are several hypotheses to be tested:  Independently of the number
+# of threads used by automake,
+# 0) the generated Makefile.in files must be identical without --add-missing,
+# 1) the Makefile.in that distributes auxiliary files must be generated
+#    after all other ones, so all installed aux files are caught,
+# 2) normal automake output should have identical content and be ordered
+#    in the same way, when --add-missing is not passed, or when
+#    --add-missing is passed but there are no concurrent file requirements
+#    (i.e., two Makefile.am files call for the same needed aux file)
+# 3) normal automake output should be identical and ordered in the same way
+#    with --add-missing, even with concurrent file requirements, and the
+#    installation of aux files should be race-free,
+# 4) warning and normal error output should be identical, in that duplicate
+#    warnings should be omitted in the same way as without threads,
+# 5) fatal error and debug messages could be identical.  This is not
+#    intended, though.
+#
+# This test checks (0).
+
+. ./defs || Exit 1
+
+set -e
+
+cat > configure.in << 'END'
+AC_INIT([parallel-am], [1.0])
+AC_CONFIG_AUX_DIR([build-aux])
+AM_INIT_AUTOMAKE
+AC_PROG_CC
+AM_PATH_LISPDIR
+AM_PATH_PYTHON
+AC_CONFIG_FILES([Makefile])
+END
+
+cat > Makefile.am << 'END'
+SUBDIRS =
+END
+
+list='1 2 3 4 5 6 7 8 9'
+for i in $list; do
+  echo "AC_CONFIG_FILES([sub$i/Makefile])" >> configure.in
+  echo "SUBDIRS += sub$i" >> Makefile.am
+  mkdir sub$i
+  echo > sub$i/Makefile.am
+done
+echo 'include foo.am' >> sub7/Makefile.am
+echo 'include bar.am' > sub7/foo.am
+echo 'python_PYTHON = foo.py' > sub7/bar.am
+echo 'lisp_LISP = foo.el' >> sub8/Makefile.am
+echo 'bin_PROGRAMS = p' >> sub9/Makefile.am
+
+rm -f install-sh missing depcomp
+mkdir build-aux
+
+$ACLOCAL
+
+# Generate expected output using the non-threaded code.
+unset AUTOMAKE_JOBS
+AUTOMAKE_run 0 --add-missing
+makefile_ins=`find . -name Makefile.in`
+for file in $makefile_ins; do
+  mv $file $file.exp
+done
+
+AUTOMAKE_JOBS=5
+export AUTOMAKE_JOBS
+
+for run in 1 2 3 4 5 6 7; do
+  AUTOMAKE_run 0
+  for file in $makefile_ins; do
+    diff $file $file.exp
+  done
+done
+
+: