Re: tail of multiple files

[Bob Proulx]

> I am using tail from the textutils 2.0 package on HP-UX 11.0.
> I have an application that runs multiple instances on the same system, each
> instance writes to a separate logfile. I then run a GNU tail job from cron 
> which
> does a tail -q -f on all the logfiles (typically 5-8) and writes them all to 
> one
> consolidated log file. The problem is that there seems to be a delay in the
> messages being logged by the tail into the consolidated log file, sometimes as
> long as 8 minutes. What seems to happen is the tail seems to write out all the
> information in one large block.

It is likely that what you are seeing is normal stdio file buffering
when the output is not a terminal.  The C library checks to see if the
output is a terminal when the program starts.  If it is then it turns
buferring off for interactive use.  If it is not a terminal then it
buffers data up into (usually) 1024 byte blocks.  These larger blocks
are much more efficient and significantly faster.  All programs that
use libc and stdio have this behavior by default.

It would be possible to add an option to turn off output buffering in
libc.

On HP-UX only there is a command called 'ied' which you might find
useful.  The purpose is to add interactive editing ala /bin/ksh to any
program as a wrapper.  Read the man page and look for the grep
example.  The ied program creates a tty on the output of the command
and therefore libc is fooled into believing that the output is a
terminal, even though I is not, and the output buffering is therefore
turned off.  At a rough guess you would use something like this.  Your
mileage may vary.

  ied -i -t tail -q -f file* > consolidatedfile

For a more portable solution you can use expect.  Here is a simple
expect script which will also create a tty around your program and
cause it to be unbuffered.  This is directly from the O'Reilly book on
expect.  You will need to have expect installed for this to work.  But
expect is generally useful and well worth the effort.

  #!/usr/bin/expect --
  # Description: unbuffer stdout of a program
  # Author: Don Libes, NIST
  eval spawn -noecho $argv
  set timeout -1
  expect

Hope that helps
Bob