bug#67455: (Record source position, etc., in doc strings, and use this in *Help* and backtraces.)

[Alan Mackenzie]

Hello, Stefan.

On Sun, Mar 10, 2024 at 13:19:03 -0400, Stefan Monnier wrote:
> > I've got a version almost ready which actually does something, namely
> > prefixes "anonymous" lines of a backtrace with the name of the defining
> > symbol, like {foo} .  It'll soon be time to start seriously thinking
> > about what information ought to go there for the live version.

> Cool!

I've finally got something to show.  I've just committed a merge and a
fix for it to branch feature/positioned-lambdas at savannah.

With this Emacs running, type the following into *scratch*

    (defun foo () "foo doc" (lambda (bar) "lambda doc" (car bar)))

..  Either evaluate this or byte compile it with compile-defun.  Then do

    M-: (funcall (foo) 'baz)

..  This will produce a backtrace like:

Debugger entered--Lisp error: (wrong-type-argument listp baz)
  car(baz)
  {foo} #f(compiled-function (bar) "lambda doc" #<bytecode 
-0x14ae78a46439bbc>)(baz)
  funcall({foo} #f(compiled-function (bar) "lambda doc" #<bytecode 
-0x14ae78a46439bbc>) baz)
  (progn (funcall (foo) 'baz))
  eval((progn (funcall (foo) 'baz)) t)
  elisp--eval-last-sexp(nil)
  {eval-last-sexp} #f(compiled-function () #<bytecode -0x1e8241efdb3d2890>)()
  eval-last-sexp(nil)
  funcall-interactively(eval-last-sexp nil)
  command-execute(eval-last-sexp)

..  Note the {eval-last-sexp} and {foo} on the anonymous functions.

:-)

> >> - Testing `byte-compile-in-progress` can't be right.  Do you to test
> >>   whether the result of this backquote will be byte-compiled or do you
> >>   really mean to test whether this backquote happens to be executed
> >>   during compilation (which could be because the compiler ends up
> >>   loading code while executing `eval-when-compile` or `require`)?

> > Quite simply, during compilation, all symbols (except nil) get read with
> > position, so to strip their positions here would be wrong.

> This isn't quite right: during compilation, some code is read with
> positions (the code that we will compile), but some code is read in the
> normal way (the code we load for the purpose of running).
> The distinction is important.

OK, I wasn't really counting code that we load as "during compilation",
but I take the point.

> >> - My gut tells me that changing backquote can't be right.

> > I tend to agree.  I put the code into backquote-process when having
> > problems with things like:

> >        (mapatoms #'(lambda (,(car spec)) ,@body)

> > in cl-macs.el, where it's impossible to know where the doc string (if
> > any) is until after the expansion of the backquotes, or even at run time
> > (as here).  In the latter case, rather than "posifying" the doc string
> > at macro expansion time, we have to generate code to do it at run time.

> Hmm... here what you call "run time" is really some later
> macro-expansion, right?  The `lambda` symbol comes from the first
> macro-expansion (ME1), but the docstring comes from the second (ME2).

Yes.  I often get confused between lots of different macro expansion
times, compile time and run time.  It's a lot easier in C.  ;-)

> IIUC the problem you face is that you want to get the function's
> position info from the `lambda` symbol, which here would be wrong (even if we
> try to preserve it long enough), is that it?

Something like that.  The lambda's position currently gets preserved in
the generated code so that ME2 can use it.

> [ Tho, in more complex cases it becomes debatable whether the function's
>   position should point to the position corresponding to ME1 or to that
>   of ME2.  ]

The code currently preserves both positions.  :-)  But only one buffer
name.

My latest thoughts on that are perhaps two file names (relative to the
Emacs top directory) would be better than one buffer name.  Then I could
put buttons on the backtrace display which on being clicked would open
either of the source files at the right position.

> More generally: what goes wrong in the above example if you just treat
> that as a list of symbol (stripping them all of their position info).
> AFAICT when *that* macro is expanded (i.e. ME2) you'll presumably get
> code like

>     (mapatoms #'(lambda (FOO/p) (DO/p SOME/p (THING/p))))

> right?  [ where "/p" means that the symbol has a sympos.  ]
> Isn't that sufficient info to add a docstring with position?

It's the lambda which has a position rather than the expanded bits from
ME2.

> >>   (lambda (f) ..) *can* appear within a backquote without it being an
> >>   actual lambda expression.
> >>   What alternatives have you considered?
> > Not a lot of them, as yet.  Maybe testing for (function (lambda ...))
> > would be safer.

> No matter how many extra tests you add to reduce the frequency, you're
> fundamentally adding a bug :-(

Yes.  I'll see what I can do to remove that extra code from
backquote-process.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).