Re: Memory again

[Dmitry Antipov]

On 12/19/2011 05:34 AM, Stefan Monnier wrote:

> `garbage-collect' is supposed to give that info.  At least it does give
> info about the data that's kept under alloc.c's control because
> of fragmentation (these are counted as "free cells").

I would like to propose a function which explicitly says how much free memory
the Emacs process holds. It's especially useful when there is a way to ask
system malloc about how much free memory is in the heap. This may be helpful
to distinguish real heap fragmentation from memory leaks and other misuses -
if the sum of values returned by 'memory-free' is, say, 10% of heap size,
then the real fragmentation enters into the game.

> Re-read what I wrote: I'm talking about the case where alloc.c free'd the
> data, but malloc did not munmap it.

I read it as 'even if we will fix allocation to use mmap and munmap, unmapped
memory would really be unused and still appear in process RSS, thus giving
higher RSS value than the process really has, until the memory pressure bump
it out of RAM'. That was obviously incorrect because RSS is recalculated
each time when mmap/mremap/munmap/brk is (successfully) called.

> It's easy to predict what it does in terms of "what is the benefit when
> I have a lot of fragmentation".  It's much more difficult to predict
> what it does in terms of "how is it going to affect Emacs's behavior for
> those 99% cases where it currently works just fine".

There is an interesting research paper about moving PLT Scheme from mark-sweep
GC to an accurate copying GC: 
http://www.cs.utah.edu/plt/publications/ismm09-rwrf.pdf.
In short, it's said about getting rid out of the most fragmentation issues at 
the cost
of 10-20% slowdown (although I consider their approach as practically 
inapplicable
beyond the pure research efforts).

> I agree that we're probably going to see better overall results by
> improving general memory use than by trying to attack
> fragmentation problems.

Among this list's subscribers, Nix <address@hidden> is constantly reporting an 
enormous
memory usage caused by Gnus. Since I newer seen such a heap usage in my own 
test cases,
I offered him a 'cumulative' patch (against 24.0.92) with both immediate 
strings and
block vector allocator. He has said that we can expect the first results after 
Christmas :-).

Dmitry

memory_free.patch
Description: Text document