Re: Is system() output guaranteed to appear in order?

[Neil R. Ormos]

Ed Morton wrote:

> I thought I remembered running an awk command
> like (simplified example of course):
> 
>    seq 5 | awk '/3/{system("echo foo")} 1'
> 
> and seeing output like:
> 
>    $ seq 5 | awk '/3/{system("echo foo")} 1'
>    1
>    2
>    3
>    4
>    5
>    foo
> 
> instead of:
> 
>    $ seq 5 | awk '/3/{system("echo foo")} 1'
>    1
>    2
>    foo
>    3
>    4
>    5
> 
> but right now I can't reproduce that and I'm
> wondering if maybe I'm misremembering or just
> haven't hit on the right scenario. Is the output
> of a call to `system()` guaranteed to always
> appear inline among the awk output at the
> location where it was called or does it depend
> on the command you're calling or some
> environment settings or something? If the
> latter, what would be an example of that?
> 
>     Ed.

system() is supposed to flush buffered output earlier written by gawk.  See 
Sec. 9.1.5 of the manual.

But it's not hard to contrive a command for system() where output from the 
system() call appears out-of-order with respect to output later written by gawk.

  seq 99999 | \
  gawk '$0==3{system("sleep 0.000000001 && echo foo &");} 1' | \
  grep -C 3 foo

Obviously you can also provoke jumbled output by writing to different output 
streams.

These artificial situations are probably not within the intended scope of the 
question, and you really just want to know if it's safe to assume output from 
the system() call will be synchronized with output printed by gawk, absent 
shenanigans.

I've had problems in the past with unsynchronized output involving 
system("command") calls when "command" doesn't directly run the program that 
generates the output, but instead sends a message asking some other facility to 
run the program or activate a new instance of it.

On Linux platforms, at least since gawk 4, I haven't noticed unsynchronized 
output from system() when regular commands are invoked, even with dd(1) which 
is said not to use standard buffering mechanisms.  I haven't tried fooling 
around with stdbuf(1), applied in various combinations to the gawk invocation 
and the command in system().  Of course, "works for me" != "guaranteed to 
always".

I hope Andy or Arnold will offer a definitive explanation.