bug#67455: (Record source position, etc., in doc strings, and use this in *Help* and backtraces.)

[Alan Mackenzie]

Hello, Stefan.

Sorry about the delay - I lost my email server after an obsolete SSL
library got deleted from my system, and one or two other things, too.

On Sat, Mar 30, 2024 at 22:54:18 -0400, Stefan Monnier wrote:
> > Some symbols must not be stripped.  For example, in cl-generic.el L403
> > we have:

> >     (fun `(cl-function (lambda ,plain-args ,@body)))              

> > ..  There the position on the lambda must be preserved until ME2 time
> > when it becomes clear what the shape of the lambda is.  In particular,
> > whether there is already a doc string in ,@body to amend, or we need to
> > insert a new one.

> I could see some reasons you *may* want to keep some info here, but it's
> definitely not a "must" because the source position of those functions
> should generally not point to `cl-generic.el:403` but to where
> `cl-defmethod` was used.

Pretending the problem doesn't exist won't solve it.  In the ;POS...
structures for a lambda, there are two pointers - one to the definition
of the lambda, the other to the point of use.

> Also, if you do want to preserve some info there (presumably with the
> intent to combine it with the more important info that will be available
> at ME2) it will need cooperation from `cl-generic.el` because, as far as
> the semantic of Emacs Lisp is concerned, the above constructs a list
> with a `lambda` symbol inside of it, with no guarantee that it will be
> used as a function, ....

Mostly there is the symbol `function' in that position.  Here we've got
`cl-function' which expands to function.  Surely with (function (lambda
....)) we know we're dealing with a function.

> .... and even if ever used as a function there's no guarantee that this
> list will pass through the few places where we strip SWPs, so keeping
> SWPs in there without some explicit request from `cl-generic.el` would
> be a bug.

I don't think this is right.  The code will pass through
macroexp--expand-all, which is where the SWP wii be stripped.

> IOW, I don't think it's a good reason to rule out

>     (strip-all-symbol-positions
>      (macroexp--expand-all
>       (read-positioning-symbols)))

As I've said, we'd need code to preserve the SWPs on "complicated"
lambdas.  I haven't even begun to think about how this could work.

> BTW, AFAIK the above is conceptually what the byte-compiler does (except
> it performs a few more transformations between `macroexp--expand-all`
> and `strip-all-symbol-positions`).

It is a bad idea to conflate these two radically different uses of SWPs.
That can only lead to confusion and bugs.

> Is it the case that `cl-defmethod` generates a function whose source
> position (partly) points to `generic.el:403` if `cl-generic.el` was
> interpreted but not if it compiled?

No, the intention is that the source positions are independent of whether
the code is compiled.

> >> >> >> Also, IIUC you don't have a separate phase to strip the SWPs when
> >> >> >> loading from source, but instead you strip them as you "consume" 
> >> >> >> their
> >> >> >> info during macroexpansion.  If so, how/when/where do you strip the
> >> >> >> false positives that may occur inside quoted data or in code like:

> >> >> >>     (defmacro foo (lambda bar) ...)

> > (defmacro foo (lambda bar)
> >   `(cons ,lambda ,bar))

> > expands to

> > (macro closure (t) (lambda bar) ";POS^^^A^A^A [foo *scratch* 158 nil]
> > " (list 'cons lambda bar))

> IIUC your reader will make the `lambda` formal argument into an SWP.
> Where is that SWP stripped?

In macroexp--expand-all in the "guard arm" near the end.

> > so it is clear this case is getting handled OK.  I'm afraid I can't
> > point out the exact place in the code at the moment where this is
> > getting done.

> I think it would be good to know, so as to be able to decide whether
> it'll indeed always work right, or we just got lucky this time.

See above.

> >> I don't actually know whether it will be better.  It just seems it could
> >> lead to simpler code, with no change at all to the reader, for example.
> > The exercise is intrinsically complicated.

> Could you explain what you think makes it intrinsically complex?

The mass of detail that needs dealing with that Emacs has collected over
the decades.  As a counter question, why do you think the exercise ought
to be simple (assuming you do think this)?

> > What you're suggesting is that the code to decide which SWPs to strip
> > is going to be simpler than the enhancements to the reader.

> As seen above, I suggest to leave the reader unchanged and to strip all
> SWPs.  I'm pretty sure it would give comparable info to what you have
> and it would be simpler (also, it would make it much less likely to
> have discrepancies between the compiled case and the interpreted case).

"Comparable" isn't good enough - we need the position info on
"complicated" lambdas to endure, somehow.  There are no discrepancies
between compiled and interpreted forms because they both use the same
mechanism in macro expansion.

> My main worry with it would be performance.

Yhat, too.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).