Re: make: *** virtual memory exhausted. Stop.

[Ted Stern]

On 07 Mar 2006 04:46:09 -0800, Mathieu Lacage wrote:
>
> hi,
>
> I am looking for a workaround to what appears to be a make 3.80 bug. A
> bit of googling turned up this:
>
> %% Fabio Alemagna <address@hidden> writes:
>
>   fa> That happens when using a construct like this:
>   fa>     define function
>   fa>     target: very long dependency list
>   fa>               whatever
>   fa>     endef
>
>   fa> and then passign that to $(eval) trough $(call). By reading the ml
>   fa> archives I got to know that this is a pretty old bug, which should
>   fa> have supposedly been fixed in a release which, at the time, was
>   fa> about to be made. However it appears that there have no been any
>   fa> more releases since then... Am I to assume that this bug will
>   fa> never be fixed (I read about the patch, but I also read it doesn't
>   fa> work as it should)?
>
> which pretty much looks like my problem. Since this email was written
> back in 2003 and no stable release of make has been made since then,
> this bug still bites some of my users who are unlucky enough to use the
> macosx-provided gnu make which does not include the gnu make patches
> used in pretty much every linux distribution under the sun (yes, my
> Makefile works nicely on every linux distribution I tried it on, most
> likely because of the patches applied by these distributions).
>
> So, I am looking for a way to work around this small limitation of gnu
> make. Is there something I could do to change the structure of my
> Makefile to avoid this ?
>
> In case someone wants to look at the whole makefile,
> http://yans.inria.fr/code/yans/ contains a mercurial repo of the
> Makefile I am using. i.e.: http://yans.inria.fr/code/yans/?
> f=0c5bb9589bbc;file=rules.mk;style=gitweb
>
>
> regards,
> Mathieu

I ran into this problem several versions ago (on various Cray UNICOS
builds).  I could generate the error when attempting to include ~10K
extra makefiles, even if their contents were empty.  Here is a script
I use to generate a bunch of subdirectories with empty files in them:

,----[ mkdirs.ksh ]
| #!/bin/ksh -p
| for dir in a b c ; do
|    i=0
|    while [[ $i -lt 10 ]] ; do
|       mkdir -p ${dir}/${dir}${i}
|       i=$((i+1))
|    done
| done
|  
| i=0
| for subdir in [a-c]/[a-c]* ; do
|    j=0
|    while [[ $j -lt 4 ]] ; do
|       touch ${subdir}/$(basename $subdir)_${i}_${j}.foo
|       i=$((i+1))
|       j=$((j+1))
|    done
| done
`----

Adjust indices/letters as necessary to create more files.

I run the script, then create a makefile with two lines:

     MKFILES := $(shell find . -type f -name '*.foo')
     include $(MKFILES)

I get the virtual memory exhausted error when about 5000 files are
included.  It depends on my shell's user limits, of course.  Since
this is an HPC environment, I run inside a PBS batch job and adjust
the job's memory limits.

It seem to recall that this occurred when configure was not able to
find a builtin alloc() function and uses a workaround.

Based on recommendations from Paul Smith, I added some "alloc(0)"
statements in various routines before return (read.c, etc).  This
frees up some memory.  But I was still not able to eliminate the
problem entirely.

Paul, do you remember this discussion?  Did you ever add a workaround?

Ted
-- 
 dodecatheon at gmail dot com
 Frango ut patefaciam -- I break so that I may reveal