Re: strange behaviour when grepping output of tail

[Bob Proulx]

> tail -f /var/log/messages | grep "foo"
> 
> above works fine and i get lines with "foo"  that gets into
> /var/log/messages.

Tail is not buffering it's output to the pipe because it is running
with the -f, follow, option.  grep is not buffering it's output
because it is running to a tty device, the stdout for your terminal.

> tail -f /var/log/messages | grep "foo" | grep "bar"
> 
> doesn't produce any output although there are lines with "foo" and "bar"

The middle grep *is* buffering.  The output there is a pipe, not a
tty, and therefore it is buffering into large block I/O.  If you
waited long enough then probably the middle grep would get enough
output gathered up that it would write a chunk all at once flushing
it's buffer of data, and then repeat the buffering process with
subsequent data.

> tail -f /var/log/messages | grep "foo" | grep "foo"
> 
> This doesn't produce any output either !!!

Same thing here.

> After breaking the tail command i test wether there were lines that should
> have been displayed. Then i see that there were valid lines
> 
> cat /var/log/messages | grep "foo" | grep "bar"
> 
> I assume it is a bug but i'm not sure

It is not a bug.  Programs use libc for stdio operations like printing
output.  The libc typically buffers it's output when being operated to
files or pipes to increase performance.  Programs can run tens or
hundreds of times faster in this mode.  But libc checks for the output
being a raw terminal and in that case only turns of buffering.

You can force buffering to be off with many contributed commands.  The
simplest is probably one the expect program comes with one called
'unbuffer'.  It is often installed with the expect package.  You run
it like this.

  tail -f /var/log/messages | unbuffer grep "foo" | grep "bar"

That wraps the process with a pty device and so makes the output look
like a tty.  However it does *use* a pty device and some systems are
limited to 60 or so of those.  It is possible to run out if you use
too many.

Hope this helps

Bob