bug#67455: (Record source position, etc., in doc strings, and use this in *Help* and backtraces.)

[Alan Mackenzie]

Hello, Stefan.

On Tue, Mar 26, 2024 at 09:40:19 -0400, Stefan Monnier wrote:
> >> > Sorry about that.  A quick summary: defined symbols (and lambda) get
> >> > positioned by the new reader function read-positioning-defined symbols.
> >> > The new declare clause defining-symbol marks a macro such as defun or
> >> > cl-defgeneric as a macro which defines such symbols.

> Since I still don't understand the general picture, let me tell you how
> I would plan to do it, so you can tell me where it matches your
> approach and where it doesn't:

> - Change `load-source-file-function` so it uses
>   `read-positioning-symbols` instead of plain `read`.
>   [ This means that macro-expansion will now almost always have sympos,
>     rather than only during compilation,  ]

load-source-file-function is set to read-positioning-defined-symbols.
(In a change to be committed, it gets bound to this function in Fload).
In reading

    (defun foo () "foo doc" (lambda (bar) "lambda doc" (car bar)))

, foo gets positioned (because it follows defun), and so does lambda
(because it is a lambda following "(").

> - This in turn requires a strip-sympos pass after the
>   eager-macroexpansion phase of `load-source-file-function`.

No such pass is needed, due to the state machine in
read-positioning-defined-symbols.

> - Change macros like `lambda` so as to use the extract position info
>   from themselves/theirargs/thecontext (when available, since there will
>   still be corner cases where it's not available, such as during
>   non-eager macro expansion) ....

The macro lambda has become obsolete; it had no access to the SWP
lambda for which it was invoked.  It has been replaced by code in
macroexpand-1 and Fmacroexpand which wraps the lambda form in (function
....) and preserves the SWP lambda.

I'm not entirely sure, but I think in non-eager macro expansion the
position information in SWPs is typically available.

>   .... and stash it in their docstring.
>   This might be as simple as adding a line
>   (setq docstring (add-pos-to-docstring docstring ARG)).

This posification is done (mainly) in a call to
byte-run-posify-lambda-form from the (function (lambda ...)) pcase arm
in macroexp--expand-all.

> - Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).