bug#67455: (Record source position, etc., in doc strings, and use this in *Help* and backtraces.)

[Alan Mackenzie]

Hello, Stefan.

On Tue, Mar 26, 2024 at 15:40:05 -0400, Stefan Monnier wrote:
> >> >> > Sorry about that.  A quick summary: defined symbols (and lambda) get
> >> >> > positioned by the new reader function read-positioning-defined 
> >> >> > symbols.
> >> >> > The new declare clause defining-symbol marks a macro such as defun or
> >> >> > cl-defgeneric as a macro which defines such symbols.

> >> Since I still don't understand the general picture, let me tell you how
> >> I would plan to do it, so you can tell me where it matches your
> >> approach and where it doesn't:

> >> - Change `load-source-file-function` so it uses
> >>   `read-positioning-symbols` instead of plain `read`.
> >>   [ This means that macro-expansion will now almost always have sympos,
> >>     rather than only during compilation,  ]

> > load-source-file-function is set to read-positioning-defined-symbols.

> [ I see it's `load-read-function`.  ]

Yes!

> How does this differ from `read-positioning-symbols` and why do we need
> it to be different?

r-p-defined-s positions only lambdas and NAMEs defined by defun,
defmacro, defvar, .... (around 50 defining symbols).  r-p-s positions
every symbol apart from nil.  They have different purposes.  r-p-d-s
gets info for the doc strings, which requires SWPs only for some
symbols.  r-p-s is needed to get warning message locations.  Were r-p-s
used for the doc string position information, most of the symbols would
need to be stripped of their positions before the form could be used.
It is simpler and faster not to position them at all.

> > (In a change to be committed, it gets bound to this function in Fload).
> > In reading

> >     (defun foo () "foo doc" (lambda (bar) "lambda doc" (car bar)))

> > , foo gets positioned (because it follows defun), and so does lambda
> > (because it is a lambda following "(").

> IIUC "gets positioned" means that it is a symbol-with-pos rather than
> a bare symbol?

Yes.

> > I'm not entirely sure, but I think in non-eager macro expansion the
> > position information in SWPs is typically available.

> In lazy macro-expansion, SWPs are not available, no.
> But that's OK, it's rare and there's very little we can do about it (the
> reason it's lazy is indeed because we only discover very late that those
> sexps were meant to represent code.  It's typically when a sexp is
> passed to `eval`).

OK.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).