[Gcl-devel] Re: re-use of streams instead of repeated fopen/flclose

[Camm Maguire]

Greetings!

Robert Boyer <address@hidden> writes:

> > But there is at least one unavoidable malloc/free per fopen/fclose which
> > appears unavoidable.
> 
> Here's my thought, perhaps more clearly stated than before.  If you have
> around a pool of streams that are already open and ready to be reused, then
> you can give one of these to the child for temporary use.  As I understand
> it, the streams (except 0, 1, and 2) in the parent are all available to and
> visible to the children.  So the parent just needs to be careful not to give
> the same temp stream out of the pool to several children at once.  When the
> child returns, the temp stream can be returned by the parent to the pool.
> Since I don't know what I am talking about really, this may well not make any
> sense at all.  I'm just trying to avoid stream creation and its costs by
> recycling them.
> 

As far as GCL is concerned, all this can be done -- even more, a
temporary tream object can be allocated on the C stack.  The problem
comes in the stdio part of libc, which GCL uses to buffer its i/o.
fopen calls malloc to allocate a FILE structure, which contains
pointers to a buffer, and integer file handle, and a bunch of other
stuff.  getc, fread, etc all are automatically buffered.  fclose then
calls free to release this structure.  malloc/free in GCL are
expensive, alas.  To get around this we have two choices 1) to define
a stream type using low-level integer file handles only and having GCL
handling the bufferring in memory of its choosing, or 2) Try to keep a
pool of file pointers returned by fopen, but then modify their
internals to refer to different integer file handles as we might need.
This latter is obviously a portability nightmare, as libc guarantees
nothing about the internal structure of FILE. 

1) is a real possibility.  Beyond this there is yet another trick --
   SYSv shared memory/ipc as a communication channel between parent
   and child.  This is the fastest known ipc available.  I've done
   this before and know how to do it.  Before getting into this level
   of complexity, however, I'd like to see how much of our existing
   overhead lies in the fork, and how much in the write/read.  If the
   bulk is in the former, no sense chasing after sysv shared memory.

Take care,

> Bob
> 
> 
> 
> 

-- 
Camm Maguire                                            address@hidden
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah