RE: "unable to open pipe" problem

[Pulley Joe Wade DR NUWCDIVNPT]

Is there a way to do this garbage collection while running? Or is this
something that has to be done within the code?  I try to "clear" all the
variables that I don't need, but that seems to make no difference
whatsoever.  When I run my script, I end up maxing out at about 190Mb of
memory usage.  When the process will no longer open a pipe, I can save the
workspace, quit and restart octave, and reload the workspace.  Everything is
present (seemingly) and the memory usage drops from 190Mb to only about 30.
So am I stuck or what?
Thanks
Dr. Joe Wade Pulley
Naval Undersea Warfare Center, Division Newport
1176 Howell Street, Building 1320, Code 3113
Newport, RI 02841
(401)-832-8690 or 1-800-669-6892 Ext. 28690


        -----Original Message-----
        From:   George White [SMTP:address@hidden
        Sent:   Wednesday, February 24, 1999 8:08 PM
        To:     Pulley Joe Wade DR NUWCDIVNPT
        Subject:        RE: "unable to open pipe" problem

        In general, if you have more free swap space than the size of octave

        you should be able to fork.  There are some problems with the stats
        presented on many systems:

        1.  fragmentation -- it may not be possible to malloc a large block
of
        memory even though the total swap space is much larger than the size
of
        the block.  This used to be problem on irix 5.3, but seems to happen
        less often irix 6.x.  If you can try your problem soon after a
reboot
        while the system is lightly loaded and it works this might be the
issue.

        2.  incomplete reporting -- some systems don't report all memory
usage,
        e.g., other programs may have "reserved" memory for copy on write
        purposes, but until the memory is actually changed, it may not be
included
        in some of the totals reported by simple tools.  On SGI there are
lots of
        optional, extra cost tools to help track such things in great detail
for
        those who need to know badly enough to spend real $$.   

        Not being able to reliably open a pipe when octave has allocated
lots of
        memory is a serious problem for my work (image processing).  As much
as
        possible, I arrange to process images in chunks, but this is more
        difficult to program and in some cases, impractical.  Matlab  
        avoids the problem by forcing me to write the data to a file (there
is
        no popen in matlab).  We use IDL quite a bit, but I haven't had time
to
        examine the way IDL handles memory.  I've been looking into R (a
stats
        language similar to S-plus that supports a reasonable set of matrix
        operations).  R uses garbage collection within a memory limit you
can
        set at startup time.  Octave requires constant attention to keep
memory
        allocation within the range where forking is reliable, while R will 
        refuse to allocate more than the limit you specified at startup.  

        --
        George White <address@hidden> Halifax, Nova Scotia

        On Wed, 24 Feb 1999, Pulley Joe Wade DR NUWCDIVNPT wrote:

        > Hi George,
        > Thanks for the reply, but I'm not sure this is the problem.  I am
not a
        > programmer, so, while I do understand the basic concept of
"forking," I
        > don't know much about the details.  However, from what you have
written, I
        > take it that this should only be a problem if I don't have enough
memory for
        > 2 copies of octave.  I assume that there is no problem with Unix
swapping
        > out one of these, so I should have enough memory since I have more
free swap
        > at the time of the problem than the size of the octave process.
Am I
        > thinking wrong here or what?
        > Thanks
        > 
        > Dr. Joe Wade Pulley
        > Naval Undersea Warfare Center, Division Newport
        > 1176 Howell Street, Building 1320, Code 3113
        > Newport, RI 02841
        > (401)-832-8690 or 1-800-669-6892 Ext. 28690




I have the same problem on SGI irix.  What happens is that octave's size
increases to the point that it can no longer 'fork' because the available
memory is not big enough for two copies.  In practice, modern systems use a
"copy-on-write" strategy so they don't actually commit the memory until the
process writes to it, so you never really run out of memory, but the system
"protects" you from yourself.  BSD provided 'vfork', which did not copy
address space, but instead allowed the 'child' process to "borrow" the
parent's space.   Note: on SGI, at least, vfork is available, but it is just
an alias for fork and does not provide BSD semantics.  
Vahallia ("Unix Internals") discusses strategies to reduce the cost of fork,
but I don't think he deals with the problem of fork in a large program, or
the extent to which vedors provide "fake" vfork implementations.  
This is one of the weak areas in the design of unix (and octave).  Some
systems allow you to specify non-existent swap space to deal with this, but
it is really a kludge for very specific situations and should probably not
be used in a general-purpose setting. 
Solaris has 'fork1' for use when the child process will quickly run a new
program.
A "modern" design strategy is to have a small "monitor" process that
controls a flock of communicating processes (octave, subshells spawned by
popen, etc.).  

--
George White <address@hidden <mailto:address@hidden> > Halifax,
Nova Scotia
On Tue, 23 Feb 1999, Pulley Joe Wade DR NUWCDIVNPT wrote:
        > Hi,
        >       I am running Octave 2.0.13 on an HP9000/750 machine running
HPUX
        > 10.2.  For the most part things seem to work fine, except when I
do things
        > that take LOTS of memory.  The calculations that use lots of RAM
seem to
        > work fine, but leave the system in a state so that if I try to do
anything
        > that opens a pipe out of octave, for example a plot or even an ls,
the
        > command fails.  The resulting message is something like "unable to
open pipe
        > for xxx" where xxx is the command.
        >       Any ideas on what may be the cause?  I am not getting errors
about
        > exhausting the memory or anything else.  Thanks
        > Wade
        > 
        > Dr. Joe Wade Pulley
        > Naval Undersea Warfare Center, Division Newport
        > 1176 Howell Street, Building 1320, Code 3113
        > Newport, RI 02841
        > (401)-832-8690 or 1-800-669-6892 Ext. 28690