bug#67455: (Record source position, etc., in doc strings, and use this in *Help* and backtraces.)

[Alan Mackenzie]

Hello, Stefan.

Thanks for taking the time and trouble to review my branch.

On Mon, Mar 25, 2024 at 14:23:50 -0400, Stefan Monnier wrote:
> >> The point is that `byte-compile-in-progress` will be non-nil during
> >> those loads, so you can't use this variable to get the information you 
> >> need.
> > Yes.  How about binding it to nil around `load' and recursive edits, and
> > possibly one or two other things?

> You mean trying to enumerate all the places we can think of where we
> know that compilation is not taking place?

Yes.

> That sounds rather ugly.  I'd rather first try and define precisely
> what is we mean by "compilation in progress".

That the byte compiler is active, and all symbols (bar nil) get
positioned.  Contrast this with, say, loading a .el file, where only some
symbols get positioned.

An alternative might be to pass an &optional boolean argument meaning
"preserve positions on symbols".

> I see the same problem with:

>       DEFVAR_LISP ("defining-symbol", Vdefining_symbol,
>              doc: /* The symbol currently being defined by a defining form.
>     This variable is bound in the read-eval-print loop and certain
>     high-level functions in the byte compiler.  It is set to a value by
>     functions and macros such as `defun', `defmacro', and `defvar'.  */);

> Lots and lots of things can happen "during the definition" of a form,
> including definition of lots of other forms.  So I think we'd need to
> define much more precisely what you meant by "currently".
> In addition, a definition is "intemporal" (it's declarative), so
> "currently being defined" is almost like an oxymoron.

By "currently", I mean that a defining form such as defun or defvar has
commenced, but not yet terminated; its functions currently occupy stack
frames.

> I'm trying to understand your code, but I clearly lack a high-level
> overview of the approach you decided to takes, so I don't understand
> what's going on there.

Sorry about that.  A quick summary: defined symbols (and lambda) get
positioned by the new reader function read-positioning-defined symbols.
The new declare clause defining-symbol marks a macro such as defun or
cl-defgeneric as a macro which defines such symbols.

The conversion of these SWPs into position structures in doc strings
happens at macro expansion time, through byte-run-posify-lambda-form.

> Is that branch trying to provide function-position for compiled
> functions only, for interpreted functions only, or both?

It not only tries, but succeeds (modulo remaining bugs) in providing
posification for both interpreted and compiled functions.

> If both, could you split it into two, then?

I'm not sure that would be possible or sensible - both use a common
approach.

> AFAICT doing it only for compiled functions should be significantly
> simpler than for interpreted functions, so it would be a good
> stepping stone.

The work has already been done, and there is working code.  Just as a
matter of interest, the branch runs the test suite without errors (not
counting "expensive" tests ).

> On the cosmetic side, you have way too much code in `byte-run.el`.
> I think most of this code can be moved elsewhere, e.g. somewhere where
> backquote can be used

Yes, I noticed this, too.  A lot of the bulk is for diagnostic functions
for SWPs, and these can eventually be deleted.  Or possibly moved into a
new file with-pos.el to be loaded before byte-run.el.

byte-run--posify-defining-symbol, the function with the extreme hand
expansion of backquotes is used as a declare clause handler, and is
needed by defun.  Hence it couldn't really be moved to after the loading
of backquote.el.

There are some additional functions which batch-posify functions and
variables defined before the posification mechanism is in place.  This
must be done ASAP, for the benefit of backtraces in early bootstrap.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).