Re: Some experience with the igc branch

[Gerd Möllmann]

Eli Zaretskii <eliz@gnu.org> writes:

>> From: Gerd Möllmann <gerd.moellmann@gmail.com>
>> Cc: Eli Zaretskii <eliz@gnu.org>,  pipcet@protonmail.com,  ofv@wanadoo.es,
>>   emacs-devel@gnu.org,  eller.helmut@gmail.com,  acorallo@gnu.org
>> Date: Fri, 27 Dec 2024 14:56:06 +0100
>>
>> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>>
>> > I've reached the point where I don't know what else to explain. Could
>> > always be improved, of course, and so on. Please find attached, with
>> > request for feedback.
>>
>> Ahem, just remembered that I had already an admin/igc.org in the branch,
>> so I've now replaced that with what I've written.
>
> Thanks.  Some questions about that file:
>
>   In contrast, the new igc collector, using MPS, is
>
>   - Concurrent.  The GC runs in its own thread.  There are no explicit
>     calls to start GC, and Emacs doesn't have to wait for the GC to
>     complete.
>
> Pip says this is not true?  I also thought MPS GC runs concurrently in
> its own thread.

What Pip said was very easy to misunderstand, to say the least :-). No,
MPS is concurrent, period. There are situations in which MPS can, in
addition, use the main thread. And it's still concurrent, period.

>   When copying objects, a marker is left in the original object pointing
>   to its copy. This marker is also called a "tombstone". A "memory
>   barrier" is placed on the original object. Memory barrier means the
>   memory is read and/or write protected (e.g. with mprotect). The
>   barrier leads to MPS being invoked when an old object is accessed.
>   The whole process is called "object forwarding".
>
> This doesn't tell how object forwarding works once triggered by access
> to protected memory.  Can you say something about that?
> This:
>
>   MPS makes sure that references to old objects are updated to refer to
>   their new addresses. Functions defined in the object format are used
>   by MPS to perform the lowest-level tasks of object forwarding, so that
>   MPS doesn't have to know application-specific details of how objects
>   look like. In the end, copying/forwarding is transparent to the
>   application.
>
> seems to try to explain that, but AFAIU stops short of telling it.
> IOW, how are "functions defined in the object format used by MPS to
> perform the lowest-level tasks of object forwarding"?
>

I'm afraid I can't describe that in detals because it's an
implementation details of MPS itself. AFAIK, it's not documented, and I
don't read the sources of MPS.

Object forwarding in general is not specific to MPS. Many copying
collectors use it. I tried to explain this as far as I could without
knowing the MPS implementation.

Whether one should describe this in detail here is a valid question.
Maybe someone knowing the MPS implementation in detail could add that.

>   AMC implements a "mostly-copying" collector, where "mostly" refers to
>   the fact that it supports ambiguous references. Ambiguous references
>   are those from ambiguous roots, where we can't tell if a reference is
>   real or not. If we would copy such an object, we wouldn't be able to
>   update their address in all references because we can't tell if the
>   ambiguous reference is real or just some random integer, and changing
>   it would have unforeseeable consequences. Ambiguously referenced
>   objects are therefore never copied, and their address does not change.
>
> This should be important to understand why some roots are submitted to
> MPS as ambiguous -- because want to prevent them from moving, right?

I don't remember ATM if we did make something an ambiguous root solely
fir the purpose of preventing to move. We have a lot of places though,
where we have to protect objects from GC, with the additional
consequence that objects don't move.

But you are right, I think.

>
>   MPS allows us to specify roots having tailor-made scan functions that
>   Emacs implements. Scanning here refers to the process of finding
>   references in the memory area of the root, and telling MPS about the
>   references.
>
> What is the purpose of "telling MPS about the references"?

If MPS were the old GC, it's like letting MPS do its mark_object.

What about if it said "... telling MPS about the reference so that it
knows the references object is live", only expressed better?

>   igc provides a number if functions for doing such allocations. For
>   example 'igc_xzalloc_ambig', 'igc_xpalloc_exact' and so on. Freeing
>   the memory must be done with 'igc_xfree'.
>
> An example of using these to reference Lisp objects in malloc'ed
> memory would be great.

OK.

>
> Stuff that I'd like added:
>
>  . a few words about the root_create_* functions
>  . same for create_*_ap functions
>  . why do we need the finalize_* functions?
>  . some explanation why pdumper needs special support from igc
>
> Thanks again for writing this.

I'll try to add something for that. Please watch out for commits.