Re: Mark procedures, LilyPond, and backward compatibility

[Mark H Weaver]

Hi Andy,

Thanks for the detailed analysis of the problems with mark procedures.

Andy Wingo <address@hidden> writes:
> At the end of our IRC conversation yesterday you put the burden on me to
> argue against mark procedures, which was fair, but at this point I think
> we would need good arguments for keeping them,

The most important reason that we need to preserve mark procedures is
that we've supported them for most of Guile's existence, and there's a
lot of important code out there that depends upon them.

Remember LilyPond?  As much as we might like to forget that LilyPond
exists and move onward with more fun things, the fact remains that as
you prepare a 2.1.x pre-release, LilyPond still doesn't work with Guile
2.0.x, and not for lack of effort.  This problem is not just going to go
away.  We _must_ help to get LilyPond working, and we must *not* take more
steps that would make that job more difficult.

We must demonstrate that Guile is a safe base upon which to build a
large application.  LilyPond is by far the most important project that
has done that, and the ongoing LilyPond+Guile2 disaster is a glaring
example that "mistakes were made" in the 1.8 -> 2.0 transition.  I think
we need to acknowledge those mistakes, learn from them, and pay more
attention to backward compatibility now and in the future.

What mistakes were made?  Among many other things: Guile 2.0 switched to
a new macro expander that doesn't support lazy macro expansion, and to a
GC with very different finalization semantics.

LilyPond has a lot of code that depends on the semantics of our old GC.
For starters, it depends heavily upon our long-standing support for mark
procedures, and will surely require ongoing support for a non-threaded
marking mode as well.

While we're on the subject: <https://bugs.gnu.org/19883> demonstrated
that Boehm GC will invoke marking procedures on objects that have
already been finalized.  It seems likely that fixing LilyPond will
require adding support to Boehm GC for another kind of finalizer with
semantics similar to the finalizers in Guile 1.8, where the finalizers
were strictly called before the mutator resumes execution.

> I think it makes sense to *avoid* adding mark functions to the new
> APIs, at least for now.  We don't have the use cases right now and so
> we would surely specify the wrong thing.  We can always add something
> later.

In summary, if we're going to deprecate and replace an existing API, the
new API needs to be designed such that it's straightforward to update
existing code to use the new API.  Therefore, we must continue to
support mark procedures.  I would go further and assert that the new API
should also include what's needed to support a precise, moving GC.  To
do otherwise would be to strongly encourage the creation of code that
cannot possibly work with a moving GC.

Deprecating and replacing the SMOB API is a significant event in the
history of Guile.  It should not be rushed, nor should it be finalized
until it supports existing use cases and anticipated future
developments.

As an aside, I'm not happy with the fact that this new foreign objects
API was pushed to the stable-2.0 branch less than 25 hours after the
initial proposal was posted.  It feels like establishing "facts on the
ground", and sets up a situation where the burden is now on me to argue
that it's not ready to be included in 2.0.12, whereas the burden should
rather be on you to obtain a consensus that this new major API is ready
for release.

For now, I would prefer to revert the commits that added this new API,
to enable the immediate release of 2.0.12.  Then we can have a proper
discussion about what a SMOB replacement API should look like, and make
sure that both past and future anticipated use cases are covered before
releasing it.

What do you think?

      Mark