Re: summary: lilypond, lambda, and local-eval

[Mark H Weaver]

I wrote:
>> Most passes of the compiler will pretend that (the-environment) is
>> replaced by tree-il code corresponding to the following standard scheme
>> code:
>
> I should also mention that perhaps, instead of simply "pretending", it
> might make sense to actually replace (the-environment) with the standard
> scheme code I gave as early as possible, so that later passes will never
> even see `the-environment' tree-il nodes.  It need only be late enough
> so that the list of visible lexical variable names is known at that
> point.

So, it turns out that the best place to transform (the-environment) into
standard scheme code is in the macro expander itself.  Indeed, only the
macro expander has enough information to generate an optimal list of
"reachable lexicals", i.e. lexical variables that are accessible using
normal symbols (as opposed to syntax objects) [more on this below].

This is great news, because it means that `the-environment' will no
longer require its own tree-il type, and the compiler will only see the
standard scheme code that it expands into.

However, we will need a robust way to either (A) specify, (B) discover,
or (C) predict the order of the captured lexicals in a closure.

Option A would be ideal, because it would allow `local-eval' to easily
and robustly create a new closure that is "compatible" with the
general-dispatcher closure created by (the-environment), using only
standard compiler interfaces.  By "compatible", I mean that their
captured lexicals have the same names and are stored in the same order
in the closure, so that code from the new closure can be spliced into
the existing general-dispatcher closure.

Option B would require `local-eval' to use a special interface to the
compiler to compile new code that is compatible with the existing
general-dispatcher closure.

Option C is similar to option B but probably more fragile, because
future changes to the compiler might invalidate the predictions and thus
break `local-eval'.  However, this fragility could perhaps be mitigated
by prominent warnings in the comments of the code that determines
closure slot order.

I have yet to decide which option to take.  Suggestions welcome.

* * * * *

There's also another issue that has come to my attention.  If we must
support arbitrary syntax-objects to be passed to `local-eval', in many
(most?) cases this will greatly increase the number of lexicals that
must be captured, and thus boxed, by (the-environment).

Consider the following example:

(let* ((t 42)
       (t (1+ t))
       (t (1+ t))
       (t (1+ t)))
  (or #f (the-environment))))

If syntax-objects are never passed to `local-eval', then only one
lexical variable is reachable from (the-environment), namely the last
`t' in the `let*', and therefore that's the only variable that must be
boxed.

However, if we must support arbitrary syntax-objects, then not only are
all four of the above `t's reachable, but the hidden `t' from the
expansion of the `or' is reachable as well!  (or at least I see no way
for the implementation of `the-environment' to prove that the user does
not possess such syntax-objects, given the way psyntax is implemented).

It would be a shame to force all of these shadowed and hidden variables
to be boxed.  If we can minimize the number of captured variables, we
can reduce the performance impact of `the-environment'.

So, I'm thinking that (the-environment) should only capture lexical
variables that are reachable using normal symbols.  I've already written
code for psyntax that generates this list of reachable lexicals.  This
should suffice in the overwhelming majority of cases.

If needed, we could easily support a variant: (the-full-environment) or
(the-environment #:full), that captures _all_ lexicals reachable using
arbitrary syntax-objects.

What do you think?

    Mark