Re: [Gcl-devel] [Maxima-discuss] gcl memory usage

[Camm Maguire]

Greetings, and thanks for the dialogue on this.

Let me state upfront that I am more than open to suggestions on how this
should be setup for our community, but I would like to avoid heuristics
and implement a rational policy even if imperfect.

I will try to write a more detailed post soon, but for now hopefully
this will clarify some things.

GCL does not allocate *anything* on startup.  Rather it probes brk() to
determine the operating system's hard limits on the data segment size.
The results of this probe are governed by the actual amount of ram+swap
physically available, the kernel's policy/algorithms (e.g. overcommit,
etc.), and the ambient system load as may be considered by the kernel.
This value is then multiplied by GCL_MEM_MULTIPLE, producing a value at
which GCL will report memory exhaustion when reached.

Thus GCL will never use up memory unless the user asks for it.  The GC
policy is then tailored to try to optimize the time/memory load tradeoff
as best as possible.

In my experience, sessions fall into two broad categories -- 1)
benchmark some production code scheduled for release, and 2) do real
incremental development work.

The rate of allocation in 2) is exceedingly slow, likely resulting in
days or weeks for a running session to begin to approach 1/2 memory.
Running speed is not at a premium, but such sessions essentially never
interfere with any other running programs anyway due to the likely small
allocation requests, so one probably wants to leave the exhaustion level
as high as possible for that occasional crazy idea, knowing that GCL
will never get there anyway for all practical purposes.

Only 1) pushes the allocation rate to meaningful levels, but here one
wants the job to finish as quickly as possible to release one's code,
finalize a package, gauge performance results, etc.  It seems
contradictory to try to run benchmarking like code on machines bogged
down with heavy browser sessions, so again one wants to be able to use
available memory as efficiently as possible.

Barring memory fragmentation issues in the implementation (an important
caveat), it would appear the only user who would want to restrict the
memory via GCL_MEM_MULTIPLE is the interactive desktop user browsing the
web and editing .doc files while running the maxima testsuite for the
sake of curiosity in the background.  It would therefore not seem
warranted to implement this as policy for everyone.

The goal of GCL's implementation is to be able to run all realistic jobs
without running out of memory with default environment settings.  Of
course it is possible that there are still corner cases where this is
not achieved and room for improvement still exists.  It is probably
impossible to eliminate these entirely with a non-copying conservative
GC algorithm, so the environment parameters are useful here as well.
That said, the current gcl debian package builds maxima on all
autobuilders with default settings at present -- the algorithm is
basically sound and robust.

When GCL_MULTIPROCESS_MEMORY_POOL is set, GCL will write a shared file
in /tmp and balance the memory among several instances.  This is so far
only critical to the ACL2 certification process.

GC is triggered based on the ratio of recent allocation to heap size.
Up to the lower limit, the trigger ratio is 100% -- i.e. GC never
happens.  The trigger ratio then declines linearly to 5%
(GCL_GC_ALLOC_MIN) as the heap grows from 1/2 available memory
(GCL_GC_PAGE_THRESH) to 3/4 (GCL_GC_PAGE_MAX) and remains fixed
thereafter.

Take care,

Gunter Königsmann <address@hidden> writes:

> On 23.04.2018 17:59, David Scherfgen via Maxima-discuss wrote:
>> Maybe you're right, Robert.
>> 
>> But I still think that Gunter may have misinterpreted GCL_MEM_MULTIPLE.
>
> I hope I didn't - even if I am not completely sure:
>
>  - In many mailing list entries GCL_MEM_MULTIPLE describes which
> fraction of the physical RAM is allocated by gcl on startup.
>  - In my measurements gcl quickly starts exceeding the limit specified
> as GCL_MEM_MULTIPLE. It seems like GCL other than SBCL is able to
> dynamically increase the amount of memory it uses after having started
> up. Which seems to be a plus for gcl.
>  - Also with the smaller GCL_MEM_MULTIPLE maxima didn't run out of
> memory in the test cases maxima-gcl with the old settings did. My guess
> is that if gcl isn't willing to allocate too much memory on start-up it
> won't fragment its memory pool as fast as with the default settings.
>  - Camm told that starting with GCL_MEM_MULTIPLE gcl's heap will grow to
> 3/4 of the physical memory installed Don't know if this amount of memory
> will get lower if GCL_MEM_MULTIPLE is decreased, though: Didn't find any
> documentation for this feature.
>
> As a side node the testbench ran impressively fast with the more
> aggressive gc settings. Normally I am used that gcl needs what feels
> like hours even for simple tasks.
>
> Where did you find documentation about GCL_MEM_MULTIPLE?
>
> Kind regards,
>
>  Gunter.
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Maxima-discuss mailing list
> address@hidden
> https://lists.sourceforge.net/lists/listinfo/maxima-discuss
>
>
>

-- 
Camm Maguire                                        address@hidden
==========================================================================
"The earth is but one country, and mankind its citizens."  --  Baha'u'llah