bug#70368: [PATCH] Use a dedicated type to represent interpreted-function values

[Eli Zaretskii]

> From: Stefan Monnier <monnier@iro.umontreal.ca>
> Cc: 70368@debbugs.gnu.org
> Date: Sun, 14 Apr 2024 09:49:23 -0400
> 
> > I don't think I understand the implications of this on compatibility
> > of byte-code.  Will the byte-code produced by Emacs 30 after these
> > changes be compatible or incompatible with previous versions of Emacs?
> 
> > And what about the compatibility in the other direction?
> 
> There are two kinds of incompatibilities it introduces, in my experience:
> 
> - "soft" incompatibilities for code which tries to display the kind of
>   the object it receives (such as in a completion table that wants to
>   add an icon indicating if something is a macro, a compiled function,
>   etc...).  Such code will still work but may display less informative
>   info because it may fail to recognize the new objects as being
>   interpreted functions.
> - "real" incompatibilities for code which digs inside the
>   entrails of functions to try and extract specific information.
>   This may fail when faced with the new interpreted functions
>   but should be easy to fix.
>   As long as such code only tries to extract info and does it via
>   `help-function-arglist`, `documentation`, and `interactive-form`,
>   there's no problem, but some packages may use code inherited from
>   a long time ago when `help-function-arglist` didn't exist, or written
>   by coders who didn't know better.  I know Hyperbole used to do that,
>   but I believe that's been fixed since.
> - "hard" incompatibilities for code which really digs inside the
>   code of functions.  `vc.el` did that a long time ago,
>   `kmacro.el` did as well until OClosures, and Buttercup (a NonGNU ELPA
>   package) did until a few months ago.  There are probably one or two
>   packages out there that will be affected like Buttercup would have
>   been.  FWIW, the Buttercup case was a mix of "hard" and "soft": it
>   really looked inside the code, but used that only to provide more
>   informative messages and had a fallback case when the code was
>   compiled, so it would still work OK.

Some of the above should be in NEWS, I think, in the "incompatible
Lisp changes" section.

> > This new #f syntax is not documented anywhere, AFAICT.  If this is the
> > new printed representation (and maybe also read syntax?) of functions,
> 
> It's only a cl-print representation, it's not `read`able.
> The `read`able representation uses the #[...] syntax also used for
> bytecode functions.
> 
> > it should be documented, like we do with other printed representations.
> 
> Where would that be?  I don't see where we document the
> #f(compiled-function ...) used for byte-code, which was my inspiration
> for the #f(lambda ...).

We have the "Printed Representation" node, and then the representation
of each type is documented where the type is documented, in
subsections of "Programming Types".  So somewhere there, I think,
probably in "Function Type"?  And you probbaly want to review
"Byte-Code Type" as well.

> >> @@ -5571,7 +5575,7 @@ display-call-tree
> >>                 " <compiled macro>"
> >>               " <macro>"))
> >>            ((eq 'lambda (car f))
> >> -           "<function>")
> >> +           "<function-like list>")
> >                   ^^^^^^^^^^^^^^^^^^^^
> > Should this be documented somewhere?
> 
> This should be an extremely rare occurrence, and `display-call-tree`
> itself is not documented, so I doubt it's worth the trouble.
> If you prefer I can keep it as `<function>`.

I thing <function> is preferable, since <function-like list> is
somewhat mysterious, and what you say above means explaining the fine
differences will be largely a wasted effort.

> >> +DEFUN ("closurep", Fclosurep, Sclosurep,
> >> +       1, 1, 0,
> >> +       doc: /* Return t if OBJECT is a function object.  */)
> >
> > If the doc string is correct, then why is the function called
> > 'closurep'?  It's against mnemonic memory.
> 
> The term "closure" is kind of funny, indeed: often it's used to say
> "this is a piece of code packaged with its environment", but in most
> cases where it's used all functions are like that, so the precise
> meaning of "closure" can be argued to be exactly the same as "function",
> just with a different connotation.  Sometimes the connotation is to
> insist on the fact that it captured some variables, but object it's used
> to distinguish between the abstract notion of functions and their
> runtime representation, where "closure" means "an object which
> represents a function".
> 
> IOW, the docstring above is meant to say "a function *object*" rather
> than "a *function* object".  Compare with `functionp`:
> 
>     Return t if OBJECT is a function.
> 
> Maybe I should say something like:
> 
>     Return t if OBJECT is a closure, i.e. a function object.
> 
> ?

Yes, I think that'd be better.

> >> +DEFUN ("interpreted-function-p", Finterpreted_function_p,
> >> +       Sinterpreted_function_p, 1, 1, 0,
> >> +       doc: /* Return t if OBJECT is an interpreted function value.  */)
> >                                                        ^^^^^^^^^^^^^^
> > "function value"? what's that?
> 
> In general, `eval` takes an source expression and returns a value.
> 
>     (lambda (x) (+ x y))
> 
> is a list of length 3 which represents a function as a source expression
> and `eval` turns it into a value (presumably a closure which will
> remember the current binding of `y`).
> 
>     (closure .. (x) (+ x y))
> 
> used to be the representation used for interpreted function *values*.

The problem is that "function value" can be interpreted as "value
returned by a function".  So I suggest

  Return t if OBJECT is a value that represents an interpreted function.

> >> +      return CALLN (Fmake_byte_code,
> >> +                    args, body, env, Qnil, docstring,
> >> +                    NILP (Fcdr (iform))
> >> +                    ? Fcar (iform)
> >> +                    : CALLN (Fvector, XCAR (iform), XCDR (iform)));
> >> +    }
> >> +  else if (!NILP (docstring))
> >> +    return CALLN (Fmake_byte_code, args, body, env, Qnil, docstring);
> >> +  else
> >> +    return CALLN (Fmake_byte_code, args, body, env);
> >
> > I'm probably missing something, but if the doc string says "make an
> > _interpreted_ closure", why does the implementation call
> > make-byte-code?  Isn't byte-code a kind-of antithesis of
> > "interpreted"?
> 
> Because `make-byte-code` is the function that we already have which
> creates PVEC_CLOSURE (previous called PVEC_COMPILED) objects.  It used
> to be used exclusively to create byte-code functions but now that same
> representation is used for interpreted function.  I could rename it but
> we already have `make-closure` for something slightly different, and
> I don't see much benefit to the rename.
> I'll add a comment explaining why we use `Fmake_byte_code`,

I think this also calls for augmenting the documentation of
make-byte-code to the effect that it could now create closures as
well.