Re: Need for scm_remember_upto_here_* in guile-2.0

[David Kastrup]

Chris Vine <address@hidden> writes:

> On Wed, 23 Sep 2015 10:54:50 +0200
> David Kastrup <address@hidden> wrote:
>> Chris Vine <address@hidden> writes:
>> 
>> > On Wed, 23 Sep 2015 09:26:27 +0200
>> > David Kastrup <address@hidden> wrote:
>> >
>> >> Also, a pointer to an array (rather than something more opaque like
>> >> SCM) is much more likely to be subject to strength reduction and
>> >> address arithmetic by the compiler, leading to a situation where
>> >> looping through some array does not leave a live pointer in
>> >> registers and stack frame in a form recognizable by libgc.
>> >
>> > However, I haven't understood your point about arrays.  If they are
>> > allocated with malloc() you need a finaliser and you may need to
>> > call scm_remember_upto_here_1() to prevent the smob's finaliser
>> > from firing while they are still in use.  If they are allocated with
>> > scm_gc_malloc() you don't, as I understand it.  Were you making a
>> > point in addition to that, and if so could you expand on it for me?
>> 
>> The point is that libgc may be less reliable with recognizing a
>> typical C/C++ array still being in use than with SCM values because
>> the compiler is more likely to mangle array access beyond recognition
>> by stack frame inspection.
>> 
>> For example, if you index arrays starting from 1 (or have indexing
>> expressions with similar effects), the actual register from which the
>> compiler does its indexing may lie strictly outside of the allocated
>> area for an array.
>> 
>> A compiler may also choose to convert a loop running from indexes 0 to
>> n-1 to one running from -n to -1 in order to have a nicer terminating
>> condition.  Again, with strength reduction this may lead to the actual
>> libgc-recognizable pointer being clear outside of the allocated array.
>> 
>> SCM is defined in a manner where the compiler has far fewer useful
>> transformations at its disposal for the normal operations.  So the
>> likelihood of libgc making mistakes is just lower.
>
> Your trees look excellent, but your pedagogical approach to the wood
> is still a mystery to me.  So are you saying:
>
> (i) libgc's garbage collector is unreliable with arrays allocated with
> scm_gc_malloc() (or scm_gc_malloc_pointerless()), so using those
> functions for garbage collected arrays should be avoided, and hand
> written memory management for arrays using smob finalisers should be
> employed instead,

Unreliable with arrays?  That's like defining what ratio of oak trees as
compared to fir trees a wood should have in order to be considered safe
from muggings at times 6am to 20pm.

Because, you know, foliage and light and density of growth.  There are
just no hard numbers.  Yet you can still pick a place where to build a
house.  Or a town.  Is a naturally protected area safe?  That's all a
matter of degrees.

> or
>
> (ii) if you are doing hand written memory management for arrays using
> smob finalisers instead of using garbage collected arrays,

That's not really "hand-written" since it still relies on conservative
stack scanning.  It's just wiring down the libgc work to the closed
subset of SCM marking and collection rather than letting it deal with
everything.  So libgc is working under better-controlled and tested
circumstances.  That relies less on the skills of the libgc programmer
(which may find a hard limit in the realities of code generation) and
more on the skills of the GUILE application programmer.

Of course, given bug #19883
<URL:http://debbugs.gnu.org/cgi/bugreport.cgi?bug=19883> it appears that
in this case it is more of a question of the libgc programmers getting
the hard part of collection right while messing up with finalizers.
Fallout from that problem is likely one of the driving factors of GUILE
developers actively recommending practices that amount to not relying on
finalizers for memory management.

GUILE would do better to mark SMOBs as dead, dead, dead (presumably by
setting the type to 0) once it calls the finalizer on it instead of
continuing to call the mark hooks (in effect, we do something comparable
in LilyPond now in order not to mark collected objects).  And/or report
this problem to the libgc authors.

Alas, "This bug report was last modified 205 days ago.".


> you should make liberal use of scm_remember_upto_here_* because libgc
> is particularly apt to finalise your smob while the arrays are still
> in use?

You should be aware what you are doing.  That's all.  One good rule of
thumb is to pass data around as SCM as long as possible and not
dereference the SCM before it is needed.  An SCM in the call frame will
usually be sufficient protection even without scm_remember_upto_here_*
as long as you don't overwrite it with other values.  Or it is passed by
register.  Which is an architecture-dependent thing so you cannot really
rely on code not needing scm_remember_upto_here_* on one platform to run
equally well on another.

No, I cannot give you hard and fast rules for woods.  I still have my
preferences, and there are reasons for them.  And there may be multiple
reasons that are mostly independent.

-- 
David Kastrup