Re: Indentation and gc

[Ihor Radchenko]

"Dr. Arne Babenhauserheide" <arne_bab@web.de> writes:

>> What is the smallest practical free RAM available to Emacs on low-end 
>> systems?
>> We can take that value and then use 800kb/min free RAM in the wild and
>> the base threshold. On system with larger RAM the threshold will scale.
>
> It’s not that simple — at least outside loading the init file. If you
> increase this fraction, then GC pauses will be longer and Emacs may feel
> jittery or even become unresponsive for some time.
>
> So there should likely be also a hard upper limit to ensure that the
> pauses are unnoticeable.

Well. I do realize that there should be a limit, which is why I put it
as 100Mb.

Strictly speaking, GC pauses scale with heap size. Increasing GC threshold
will have two effects on the heap size: (1) thresholds lager than normal
heap size will dominate the GC time - Emacs will need to traverse all
the newly added data to be GCed; (2) too large thresholds will cause
heap fragmentation, also increasing the GC times as the heap will expand.

I think that (2) is the most important factor for real world scenarios
(unless we set the threshold insanely high, higher than memory usage in
"heavy" Emacs sessions).

Emacs' default gives some lower safe bound on the threshold - it is
`gc-cons-percentage', defaulting to 1% of the heap size.

However, being safe is unfortunately not enough - apparently 1% heap
size is too less in real world scenarios when 1% is routinely and
frequently allocated, triggering GC rescans too many times.

Especially so, when loading init.el. The heap size is yet smaller than
normal and GC is triggered even more frequently.

AFAIU, routine throw-away memory allocation in Emacs is not directly
correlated with the memory usage - it rather depends on the usage
patterns and the packages being used. For example, it takes about 10
complex helm searches for me to trigger my 250Mb threshold - 25Mb per
helm command. The GC frequency often depends on how heavily I use helm
completion.

To get some idea about the impact of gc-cons-threshold on memory
fragmentation, I compared the output of `memory-limit' with 250Mb vs.
default 800kb threshold:

 250Mb threshold - 689520 kb memory
 800kb threshold - 531548 kb memory

The memory usage is clearly increased, but not catastrophically, despite
using rather large threshold.

Of course, it is just init.el, which is loaded once. Memory
fragmentation as a result of routine Emacs usage may cause more
significant memory usage increase. Not multiple times though (not even
twice in my setup).

>From the above, I expect the increase of gc-cons-threshold 100x to
impact the memory dis-proportionally - between as little as 1.1x and 2x.
As long as gc-cons-threshold is significantly (~10x) lower than normal
Emacs heap size, I expect the GC frequency to scale down the same 100x
at very little cost.

For me, Emacs memory usage typically settles around 1Gb, which is why I
chose 100Mb as an upper limit (1Gb/10).

If we want to be even more safe, as I proposed, we can only increase the
gc-cons-threshold when loading init.el and possibly for certain, most
GC-heavy commands. This will minimize the impact of memory
fragmentation - only a small set of commands (having more regular memory
allocation pattern) will cause the fragmentation.

-- 
Ihor Radchenko // yantar92,
Org mode contributor,
Learn more about Org mode at <https://orgmode.org/>.
Support Org development at <https://liberapay.com/org-mode>,
or support my work at <https://liberapay.com/yantar92>