Re: local-eval on syntax-local-binding, bound-identifiers

[Mark H Weaver]

Hi Andy!

Andy Wingo <address@hidden> writes:
>>> +                     (cons (wrap (car symnames)
>>> +                                 (anti-mark (make-wrap (car marks) subst))
>>
>> ***** Why are you adding anti-marks here?
>
> As the changelog noted (and a comment should have noted ;), the
> identifiers are anti-marked so that syntax transformers can introduce
> them, as-is.
>
> The purpose of this procedure is to get a list of identifiers, and to
> capture some subset of them.  It will do so by introducing references to
> them in the expansion of some macro.  However they are not introduced
> identifiers: they come from the code itself.  They are input the macro,
> and as such need an anti-mark.
>
> The anti-mark will be stripped from the expansion when the transformer
> that called `bound-identifiers' returns.

Does this mean that `bound-identifiers' will not function properly when
used outside of a macro?  What about if it's used within a macro that
was generated by another macro (or things of that nature)?  Are there
cases where you might need to strip more than one anti-mark?

To use your phrase, this has a bad smell.

>> More importantly: I notice that you are not stripping the psyntax wrap
>> from identifiers placed within the wrapper procedure above.  There are
>> certainly benefits to that, but remember that the wrapper procedure will
>> in general be serialized to disk and evaluated in a different Guile
>> session, where the gensym counters have been reset.
>
> Of course, like all macros!  The forgeable gensym issue is something we
> have in Guile, more generally, that needs a broader solution.

Ah, good point!  Macros already serialize syntax-objects to disk.
psyntax wraps are already part of our ABI, so nothing new there.

However, I fear that the gensym issue might be a serious problem for
`local-eval', even though it hasn't been a problem for macros.

The reason it has not been a problem with macros is that, within a
top-level macro (which are the only ones used across Guile sessions),
the only syntax-objects that can be meaningfully _introduced_ into the
expansion are top-level/module bindings.  But these bindings have no
associated labels or gensyms, because they're not in the wrap.

On the other hand, with `local-eval', it seems to me quite plausible
that gensym collisions might occur.  Suppose in one Guile session you
compile a procedure (foo) that uses (the-environment), and then in
another Guile session, you call (foo) and then `local-eval' with the
environment returned by (foo).  Now the wrapper procedure splices
together syntax objects from two different Guile sessions into a single
top-level form, where (unlike in the macro case) all of these syntax
objects are lexicals, and thus depend on the gensyms and the labels.

See how this is a problem now where it wasn't before?
Or am I missing something?

>>> +        ((module? e)
>>> +         ;; Here we evaluate the expression within `lambda', and then
>>> +         ;; call the resulting procedure outside of the dynamic extent
>>> +         ;; of `eval'.  We do this because `eval' sets (current-module)
>>> +         ;; within its dynamic extent, and we don't want that.  Also,
>>> +         ;; doing it this way makes this a proper tail call.
>>> +         ((eval #`(lambda () #,x) e)))
>>
>> ***** This was my mistake, but since I'm already marking up the code:
>> the `lambda' wrap above needs a `#f' before `e' to force expression
>> context.
>
> OK.  (Note though that (eval X e) does indeed evaluate X in tail
> position.)

Looks to me like `eval' is initially bound to the C function scm_eval.
Is it later rebound to a Scheme procedure?  If so, where?

>> For the record, I still think it's better for `the-environment' to be
>> implemented within psyntax as a core form.  It's a fundamental syntactic
>> construct with clean semantics, and it belongs in psyntax with its
>> brethren.  Your desire to remove it from psyntax has caused you to add
>> far less elegant interfaces that have been hastily designed, and that
>> may not even be sufficient for a full implementation of
>> `the-environment' that captures mutually-recursive local macros.
>
> In pursuit of the goal of agreeing on a strategy, I would like to
> convince you that you are wrong on all of these points :)  So, in that
> spirit, I argue:

Very well, I will endeavor to be open-minded.

> `the-environment' is not fundamental: it can be implemented in terms of
> simpler primitives.

The same can be said of `lambda' or `syntax-case', but that's not the
appropriate way to choose primitives in a language.  The set of
primitives chosen in Scheme are not the ones that are simplest to
implement.  It makes more sense to choose primitives with simple, clean
semantics, which segues nicely into your next paragraph.

> `the-environment' does not have clean semantics, inasmuch as it has
> nothing worthy of the name, not yet anyway.  The lambda calculus, the
> scheme language, even the syntax-case system have well-studied semantics
> (denotational and/or operational), and lots of experience.  The same
> cannot be said of `the-environment'.  It is great that we are pushing
> the boundaries of Scheme here, but I think it's hubris to think that
> we'll get it right the first time.

I think you read more into my words "clean semantics" than I intended.
Of course I never meant to imply that its semantics are as well-studied
as the rest of standard Scheme.  I only meant to express my own personal
judgement that the semantics of `the-environment' and `local-eval' seem
quite clean.

BTW, did you see my most recent model for thinking about `local-eval'?

(the-environment) expands to (list (lambda () <expr>) ...), with one
element for every possible expression: a countably infinite list that
could be built lazily.  `local-eval' simply chooses the appropriate
procedure from the list and calls it.  A poor implementation strategy,
but the semantic meaning is quite clear, no?

Regardless, I hope it's safe to say that `the-environment' and
`local-eval' are far closer to the spirit of primitives one finds in the
Scheme standards than are the ones you're proposing.  Do you not agree?

> Implementing the-environment in psyntax also poses implementation
> layering problems: it uses wrap hacks to expand out to private forms
> from ice-9 local-eval.

I don't see how that's a "hack" at all.  That's exactly what wraps are
for: to specify the expander's lexical environment in which to look up
the symbol.  I'm using them in a completely straightforward way to
create syntax-objects for identifiers in ice-9/local-eval.  This within
psyntax, where wraps are created all over the place.

> It also seems to me that stripping identifiers down to their symbolic
> values is the wrong thing to do.  It has a bad smell.

I agree 100%.  Keeping the entire syntax-objects is "the right thing",
and indeed I think we'll need this in order to capture local macros.

However, making this work properly requires universally unique gensyms.

I decided to avoid this complication for now, because it seems to me
that it's highly unlikely to matter in practice until we can capture
local macros.  However, if you'd like to fix the gensym problem for
2.0.4, that's even better.

> I think it's possible that this approach could work for mutually
> recursive macros.  The key to note here is that the essence of the
> mutual recursion is in the expansion of the syntax transformers (the
> rhs), and that was already done within a proper recursive environment.

That only works if we can reuse the transformer procedures that were
originally captured by (the-environment).  And indeed, this is the only
way to capture _procedural_ macros properly.  Therefore, I have no hope
that such macros could be serialized to disk.

`syntax-rules' macros are a different story.  A `syntax-rules' macro can
be _perfectly_ represented as a syntax-object of the `syntax-rules' form
itself.  Therefore, I would like to represent them this way in the
expansion of (the-environment), so that `syntax-rules' macros _can_ be
serialized to disk.

However, this requires re-expanding the bodies of the `syntax-rules'
forms, so now the environment structures _do_ matter.  Ultimately, this
means that a set of mutually recursive local macros must be recreated in
`local-eval' as a `letrec-syntax'.

Does that make sense?

    Thanks,
      Mark