parallel make and I/O of child processes

[Stefano Lattarini]

Hello autoconfers and automakers.

In both FreeBSD and NetBSD make, when "parallel mode" is used (e.g., with
`make -j2'), the stdout and stderr of child processes get connected to
pipes[1][2].  Similarly, in Solaris 10 dmake, the stdout and stderr of
child processes get connected to temporary files[3].

IMHO, the developers of a software package should be fully aware of this
behaviour when writing their makefiles, since it might cause some small
yet potentially subtle semantic differences in the behaviour of the
spwaned processes.

Here is a concrete example of how such semantic differences can bite:
the "colored output" from automake-generated testsuites[4] doesn't work
out-of-the box with NetBSD and FreeBSD make run in "parallel mode",
even when those make instances are connected to a capable terminal[5][6].

So, what about the attached documentation patches (one for the automake
maint branch, the other for the autoconf master branch)?  Do you think
they could be useful?

Regards,
   Stefano

-*-*-


"FOOTNOTES"


 [1] 
    I *think* this is done in order to avoid to intermingle output
    from different processes, to some degree at least.  For example:

      $ cat Makefile
      all: a b
      a:
          @sleep 0.1; printf a; sleep 0.1; printf a; sleep 0.1; echo
      b:
          @sleep 0.15; printf b; sleep 0.1; echo b
      $ gmake -j2
      abab

      $ heirloom-make -j2
      abab

      $ freebsd-make -j2
      bb
      aa
      $ netbsd-make -j2
      --- a ---
      --- b ---
      bb
      --- a ---
      aa


 [2]
    This can be demonstrated (on Linux) with the following:

      $ echo 'all: ; ls -l /proc/$$$$/fd' > Makefile

      $ gmake -j 2
      ls -l /proc/$$/fd
      total 0
      lrwx------ 1 stefano stefano 64 Jan 24 22:49 0 -> /dev/pts/4
      lrwx------ 1 stefano stefano 64 Jan 24 22:49 1 -> /dev/pts/4
      lrwx------ 1 stefano stefano 64 Jan 24 22:49 2 -> /dev/pts/4
      lr-x------ 1 stefano stefano 64 Jan 24 22:49 3 -> /proc/14727/fd

      $ heirloom-make -j 2
          ls -l /proc/$$/fd
      total 0
      lrwx------ 1 stefano stefano 64 Jan 24 22:50 0 -> /dev/pts/4
      lrwx------ 1 stefano stefano 64 Jan 24 22:50 1 -> /dev/pts/4
      lrwx------ 1 stefano stefano 64 Jan 24 22:50 2 -> /dev/pts/4

      $ netbsd-make -j 2
      --- x ---
      ls -l /proc/$$/fd
      total 0
      lrwx------ 1 stefano stefano 64 Jan 24 22:50 0 -> /tmp/make4lRgpl 
(deleted)
      l-wx------ 1 stefano stefano 64 Jan 24 22:50 1 -> pipe:[34633]
      l-wx------ 1 stefano stefano 64 Jan 24 22:50 2 -> pipe:[34633]

      $ freebsd-make -j 2
      ls -l /proc/$$/fd
      total 0
      lrwx------ 1 stefano stefano 64 Jan 24 22:53 0 -> /tmp/makeXXXXAYm9oW 
(deleted)
      l-wx------ 1 stefano stefano 64 Jan 24 22:53 1 -> pipe:[34818]
      l-wx------ 1 stefano stefano 64 Jan 24 22:53 2 -> pipe:[34818]


 [3]
     This can be demonstrated (on Solaris 10) with the following:

       $ echo 'all: ; all: ; /usr/proc/bin/pfiles $$$$' > Makefile

       $ gmake -j 2
       Current rlimit: 256 file descriptors
         0: S_IFCHR mode:0620 ...
            O_RDWR|O_NOCTTY|O_LARGEFILE
            /devices/pseudo/address@hidden:6
         1: S_IFCHR mode:0620 ...
            O_RDWR|O_NOCTTY|O_LARGEFILE
            /devices/pseudo/address@hidden:6
         2: S_IFCHR mode:0620 ...
            O_RDWR|O_NOCTTY|O_LARGEFILE
           /devices/pseudo/address@hidden:6

       $ dmake -j 2
       Current rlimit: 256 file descriptors
        0: S_IFCHR mode:0620 ...
           O_RDWR|O_NOCTTY|O_LARGEFILE
           /devices/pseudo/address@hidden:6
        1: S_IFREG mode:0600 ...
           O_WRONLY|O_DSYNC|O_CREAT|O_TRUNC|O_LARGEFILE
           /tmp/dmake.stdout.23631.0.7ba4jU
        2: S_IFREG mode:0600 ...
           O_WRONLY|O_DSYNC|O_CREAT|O_TRUNC|O_LARGEFILE
           /tmp/dmake.stdout.23631.0.7ba4jU


 [4]
    For more details, see the description of the 'color-tests' option at:
    <http://www.gnu.org/software/automake/manual/html_node/Simple-Tests.html>


 [5]
    That happens becuse the automake-generated testsuite drivers check
    whether the controlling make instance is attached to a tty (to decide
    whether to activate colored output or not) by using `test -t 1' in
    a proper recipe.  But when make is run in "parallel mode", the shell
    running that test has its stdout connected to a pipe (read my make),
    so that the test returns "false", and the colored output is not
    activated.


 [6]
    Luckily, there is a simple work around: define the AM_COLOR_TESTS
    variable to 'always' in the environment of the make process; e.g.,
    the following will produce colored testsuite output even when
    using NetBSD or FreeBSD make:
      $ AM_COLOR_TESTS=always make -j2 check

From 0f216c0ad0ab721e25c0d16b9fb66fbcb1a5ae64 Mon Sep 17 00:00:00 2001
From: Stefano Lattarini <address@hidden>
Date: Mon, 24 Jan 2011 22:31:37 +0100
Subject: [PATCH] docs: another parallel make issue

* doc/autoconf.texi (Parallel Make): Document that some make
implementations, when run in parallel mode, connect stdout and
stderr of child processes to pipes or temporary files.
---
 ChangeLog         |    7 +++++++
 doc/autoconf.texi |   37 ++++++++++++++++++++++++++++++++++++-
 2 files changed, 43 insertions(+), 1 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index 2fc5ab3..e548f8c 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,10 @@
+2011-01-24  Stefano Lattarini  <address@hidden>
+
+       docs: another parallel make issue
+       * doc/autoconf.texi (Parallel Make): Document that some make
+       implementations, when run in parallel mode, connect stdout and
+       stderr of child processes to pipes or temporary files.
+
 2011-01-23  Ralf Wildenhues  <address@hidden>
 
        docs: new section about whitespace trimmmed from make command-lines.
diff --git a/doc/autoconf.texi b/doc/autoconf.texi
index ae601a0..eaeb054 100644
--- a/doc/autoconf.texi
+++ b/doc/autoconf.texi
@@ -19475,7 +19475,42 @@ sh
 @cindex Parallel @command{make}
 
 Support for parallel execution in @command{make} implementation varies.
-Generally, using GNU make is your best bet.  When NetBSD
+Generally, using GNU make is your best bet.
+
+Some make implementations (among them, FreeBSD @command{make}, NetBSD
address@hidden, and Solaris @command{dmake}), when invoked with a
address@hidden@var{N}} option, connect the standard output and standard
+error of all their children processes to pipes or temporary regular
+files.  This can lead to slightly but subtly different semantics in
+the behaviour of the spawned processes.  For example, even if the
address@hidden standard output is a connected to a tty, a command
+like @code{test -t 1} issued in a recipe will return a failure within
+those make implementations:
+
address@hidden
+$ @kbd{cat Makefile}
+all:
+        @@test -t 1 && echo "Is a tty" || echo "Is not a tty"
+$ @kbd{make -j 2} # FreeBSD make
+Is not a tty
+$ @kbd{make -j 2} # NetBSD make
+--- all ---
+Is not a tty
+$ @kbd{dmake -j 2} # Solaris dmake
address@hidden --> 1 job
address@hidden --> Job output
+Is not a tty
address@hidden example
+
address@hidden
+while on the other hand:
+
address@hidden
+$ @kbd{make -j 2} # GNU make, Heirloom make
+Is a tty
address@hidden example
+
+When NetBSD
 @command{make} is invoked with @address@hidden, it will reuse the
 same shell for multiple commands within one recipe.  This can have
 unexpected address@hidden that GNU make has
-- 
1.7.2.3

From 3eaf9247e44a354cc0c0606e2c0f2b564e4c1c4e Mon Sep 17 00:00:00 2001
From: Stefano Lattarini <address@hidden>
Date: Mon, 24 Jan 2011 22:26:55 +0100
Subject: [PATCH] docs: color-tests issues with parallel make

* doc/automake.texi (Simple Tests): Document that automatic
recognition of a capable terminal attached to stdout can fail
with some make implementation when running in parallel mode,
thus causing colored test output not to be automatically
activated when it should.
---
 ChangeLog         |    9 +++++++++
 doc/automake.texi |    6 ++++++
 2 files changed, 15 insertions(+), 0 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index c632b44..95790da 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,12 @@
+2011-01-24  Stefano Lattarini  <address@hidden>
+
+       docs: color-tests issues with parallel make
+       * doc/automake.texi (Simple Tests): Document that automatic
+       recognition of a capable terminal attached to stdout can fail
+       with some make implementation when running in parallel mode,
+       thus causing colored test output not to be automatically
+       activated when it should.
+
 2011-01-23  Stefano Lattarini  <address@hidden>
 
        coverage: test semantics of "dummy" per-target flags
diff --git a/doc/automake.texi b/doc/automake.texi
index 3d6a7b0..2391621 100644
--- a/doc/automake.texi
+++ b/doc/automake.texi
@@ -8589,6 +8589,12 @@ colored output by setting the @command{make} variable
 @samp{AM_COLOR_TESTS=no}, or force colored output even without a connecting
 terminal with @samp{AM_COLOR_TESTS=always}.
 
+Note that incompatible semantics of some @command{make} implementations
+when used in parallel mode (@pxref{Parallel make,,, autoconf, The Autoconf
+Manual}) can cause the automatic detection of a connection to a capable
+terminal to fail spuriously.  In that case, you can still resort to the
+use of @samp{AM_COLOR_TESTS=always}.
+
 @vindex TESTS
 @vindex TESTS_ENVIRONMENT
 The variable @code{TESTS_ENVIRONMENT} can be used to set environment
-- 
1.7.2.3