[Groff] pre-grohtml fails with SIGPIPE on command line invocation

[MARSHALL Keith]

Werner, Gaius,

When I submitted my original patch set, to add Win32 capability to
'pre-grohtml', you raised the following issue, Werner:

>> 12.     modified src/preproc/html/pre-html.cc to use Win32 semantics
>>         for spawning subprocesses, when compiled for a Win32 target;
>>         this continues to work as before, when built for a UNIX target;
>
> No, it doesn't, unfortunately.  Running e.g.
>
>   pre-grohtml troff -ms -Thtml pic.ms
>
> it crashes with SIGPIPE. 

I have finally had a chance to analyse this behaviour, and am fairly
confident that I now understand what causes it.

> Please fix this -- my knowledge of pipes is nonexistent.

Ok. There is nothing particularly complicated or mysterious about
them.  Basically:

  = Assuming a declaration ...
 
      int status, filedes[2];
 
    a process may create an unnamed pipe, by calling the ...

      status = pipe( filedes );

    service function; the kernel responds by initialising a pair of
    I/O streams, and returns file descriptors for each, in the
    'filedes' array. (If this fails, 'pipe' returns a 'status' value
    of -1;  a real example should check this, and provide appropriate
    error handling.)

  = The process can now write data sequentially to 'filedes[1]', and
    read this same data back, in the same sequence, from 'filedes[0]';
    essentially, the two file descriptors may be used exactly the same
    way as descriptors for regular files, provided a few caveats and
    restrictions are observed:

      - The only operations permitted on the associated I/O streams
        are sequential reading and writing, for 'filedes[0]' and
        'filedes[1]' respectively, and, ultimately, closing of both
        descriptors; any attempt to rewind the stream, or to 'seek' to
        any nonsequential position in the stream, will fail.

      - Duplicate descriptors for either stream may be created, either
        explicitly by 'dup'/'dup2', or implicitly, as a result of
        inheritance by a child process.

      - Reading from 'filedes[0]', or any duplicate of it, will block
        until data has been written to 'filedes[1]' or any of its
        duplicates; after data has been written, reading will retrieve
        it, up to the number of bytes actually written; subsequent
        reads, after all written data has been retrieved will again
        block.

      - Reading from 'filedes[0]' after 'filedes[1]' and all of its
        duplicates have been closed will return any written data which
        has not yet been read; thereafter, 'filedes[0]' will report an
        end-of-file condition.

      - Writing to 'filedes[1]' after 'filedes[0]' and all of its
        duplicates have been closed is not permitted; this is the
        cause of the SIGPIPE exception.

  = Creating a pipe for use only by a single process is much like
    talking to oneself, and is not particularly useful.  Therefore,
    the invocation of 'pipe' is normally followed by a 'fork' and
    'exec' (in UNIX), or 'spawn' (in Win32).  The child process, thus
    created, inherits duplicates of the two file descriptors, and
    therefore, also inherits the capability to both write data to, and
    read data from, the pipe; the pipe is thus established as a
    conduit for passing data between the parent and child processes.

  = While it is probably technically possible to use a single pipe for
    bidirectional data transfer, the synchronisation logic would be
    horrendously complex;  therefore, it is normal to treat any pipe
    as a one way data conduit.  Thus:

      - One of the two processes, (the child, in the pre-grohtml
        context), becomes the reader, using only the descriptor
        associated with 'filedes[0]'; if practicable, (as it always is
        in the UNIX 'fork'/'exec' scenario, but may not be for a Win32
        child process invoked by 'spawn'), it should immediately close
        its descriptor associated with 'filedes[1]', before starting
        to read the pipe data, and ultimately closing the descriptor
        associated with 'filedes[0]', only after the end-of-file
        condition is detected.

      - The other process, (the parent, in the pre-grohtml context),
        becomes the writer, using only the descriptor associated with
        'filedes[1]';  it has no use for 'filedes[0]', and therefore,
        if possible, should close it soon after the 'fork' or 'spawn'
        call.  It then writes data, as required, to the descriptor
        associated with 'filedes[1]', and ultimately closes this
        descriptor, to tell the reader there is no more data to come.

Now, if I go back to the source for pre-grohtml version 1.18.1, I see
two remarkably similar functions, 'do_html' and 'do_image', both of
which appear to implement the above, but only in respect of the UNIX
'fork'/'exec' method for invoking the pipe reader process.  Version
1.19.1 addresses this, merging much of the commonality of these two
functions into the single function, 'run_output_filter', and adding
support for the Win32 'spawn' method of invoking the reader;
'run_output_filter' implements the pipe writer/reader functionality,
exactly as described above.

On cursory inspection, it appears that both versions of pre-grohtml
should behave the same, at least in respect of the pipe reading and
writing functionality required by 'do_html' and 'do_image'.  However,
as Werner has observed, running, say ...

    pre-grohtml troff -ms -Thtml webpage.ms

in the 'doc' directory of the groff distribution, crashes with SIGPIPE
for version 1.19.1, but "hangs" for version 1.18.1.  (I have chosen
'webpage.ms' for my example, rather than 'pic.ms', simply because
'webpage.ms' is smaller, and does not require preprocessing by 'pic').

This puzzled me;  I expected identical behaviour between the two
versions, and clearly, we were seeing something else.  It has taken a
fair bit of debugging effort, but finally, I have identified the
difference -- version 1.18.1 never closes 'filedes[0]' in its writer
process implementation of 'do_image', whereas version 1.19.1 closes it
*immediately* after forking the reader process!  (Version 1.18.1's
'do_html' *does* close 'filedes[0]', as 'run_output_filter' does;  I
didn't notice the difference in 1.18.1's 'do_image', when I was
implementing 'run_output_filter').

So, here is what actually happens, when we run ...

    pre-grohtml troff -ms -Thtml webpage.ms

'pre-grohtml' parses the command arguments, and performs some
initialisation steps, before eventually reaching 'do_image', where it
creates a pipe, and 'exec's ...

    groff -ms -Tps webpage.ms -rps4html=1 \
          -dwww-image-template=grohtml-10280 -P-pletter
 
as the reader process.  This completes very quickly -- 'groff' doesn't
realise that we really wanted it to read data from 'pre-grohtml's
pipe, and reads 'webpage.ms' instead, as it has been told to -- so the
reader dies long before 'pre-grohtml' has a chance to write the image
data.

Now, 'pre-grohtml' does not know that its reader process has died, and
continues to write the image data to the pipe.  For version 1.19.1,
the kernel catches this, because there is no reader for the pipe, and
*correctly* sends 'pre-grohtml' a SIGPIPE, which it must either
handle, (which it doesn't), or abort, (which it does).

In the case of version 1.18.1, there is again no reader for the pipe,
but the kernel doesn't know this --  because 'pre-grohtml' neglects to
close 'filedes[0]' in the writer process, the kernel expects
'pre-grohtml' itself to read the pipe, and so, does not raise the
SIGPIPE;  'pre-grohtml' blithely writes pipe data until the kernel's
buffer is full, then blocks until something reads some of the data, to
make room for more.  Nothing ever does read this data, so 'pre-grohtml'
"hangs" with a deadlock contention for the pipe, until it gets SIGQUIT
or SIGKILL to abort it.

It has taken a while to get here, but I hope all of the above is
relevant!  Werner, I can easily modify 'run_output_filter' to emulate
the behaviour of version 1.18.1.  However, given the above analysis, I
am not sure that I should -- I think that that would be a case of
fixing a symptom of a problem, rather than fixing the problem itself!
As I see it, the *real* problem here is not that 'pre-grohtml' crashes
on SIGPIPE, when invoked in this somewhat contrived manner, but rather
that 'pre-grohtml' does not correctly process files which are
specified as file name arguments, passed to it on the command line,
(and as its own 'usage' description suggests that it should).

There are, I think, three options here:

  1. Modify 'run_output_filter', so that SIGPIPE is ignored;  (this is
     easy, but I am reluctant to do it, since I do not believe it to
     be the most appropriate action).

  2. Declare that 'pre-grohtml' should *always* process 'stdin', and
     *never* files specified as command line arguments;  modify the
     text displayed by the 'usage' function, and the parsing of
     command line arguments accordingly, such that 'pre-grohtml' will
     abort with a syntax error message, if file arguments are
     specified;  (this is probably also fairly easy, but likely to
     require a bit more effort than 1).

  3. Modify the operation of 'pre-grohtml', such that files specified
     on the command line are processed appropriately.  Modification of
     the argument parsing code will also be required; at the very
     least, it will be necessary to eliminate such file names from the
     argument lists passed to 'do_image' and 'do_html';  (perhaps this
     should be the ultimate goal, but is likely to require
     significantly more effort than either 1 or 2).

What do you think?

Best regards,
Keith