bug#67455: (Record source position, etc., in doc strings, and use this in *Help* and backtraces.)

[Alan Mackenzie]

Hello, Stefan.

Thanks for looking at my patch!

I've got a version almost ready which actually does something, namely
prefixes "anonymous" lines of a backtrace with the name of the defining
symbol, like {foo} .  It'll soon be time to start seriously thinking
about what information ought to go there for the live version.

Unfortunately, I've still got two blocking issues to resolve - one is a
change in functionality to Fbare_symbol, which I've reported as a bug;
the other is some new code merged in from master, a with-eval-after-load
clause in pcase.el which causes havoc in my build.

Still, progress is being made.

On Sat, Mar 09, 2024 at 16:36:57 -0500, Stefan Monnier wrote:
> > I've just pushed a large commit to feature/positioned-lambdas, a work
> > in progress commit for bug#67455, putting source position information at
> > the start of doc strings.  master was merged into it just before the
> > commit.

> I barely started to look at the code, but regarding the following hunk:

>     diff --git a/lisp/emacs-lisp/backquote.el b/lisp/emacs-lisp/backquote.el
>     index 6917128d70a..0d4681bc979 100644
>     --- a/lisp/emacs-lisp/backquote.el
>     +++ b/lisp/emacs-lisp/backquote.el
>     @@ -172,6 +172,30 @@ backquote-process
>            (backquote-delay-process s (1- level))))
>         ((eq (car s) backquote-backquote-symbol)
>            (backquote-delay-process s (1+ level)))
>     +   ;; Process a (lambda ...) form specially, since otherwise the
>     +   ;; lambda symbol would get separated from its introductory (,
>     +   ;; preventing this processing from being done elsewhere in macro
>     +   ;; expansion.
>     +   ((and (eq (car s) 'lambda)
>     +         (symbol-with-pos-p (car s))
>     +         (listp (car-safe (cdr s))))
>     +    (let ((kdr (backquote-process (cdr s) level))
>     +          (lambda-pos (symbol-with-pos-pos (car s)))
>     +          )
>     +      (if (null byte-compile-in-progress)
>     +          (setcar s 'lambda))           ; Strip the position.
>     +      (cond
>     +       ((= (car kdr) 0)
>     +        (cons (car kdr)
>     +              (list 'quote
>     +                    (byte-run-posify-lambda-form
>     +                     (cons (car s) (car (cdr (cdr kdr)))) ; Two cdr's to 
> strip 'quote.
>     +                     lambda-pos))))
>     +       (t
>     +        (cons 1
>     +              (list 'byte-run-posify-lambda-form
>     +                    (list 'cons (list 'quote (car s)) (cdr kdr))
>     +                    lambda-pos))))))
>         (t
>          (let ((rest s)
>               item firstlist list lists expression)

> - Testing `byte-compile-in-progress` can't be right.  Do you to test
>   whether the result of this backquote will be byte-compiled or do you
>   really mean to test whether this backquote happens to be executed
>   during compilation (which could be because the compiler ends up
>   loading code while executing `eval-when-compile` or `require`)?

Quite simply, during compilation, all symbols (except nil) get read with
position, so to strip their positions here would be wrong.  Outside of
compilation (an evaluation of a defun, for example), only defined
symbols get positioned, and these symbols with position would interfere
with Emacs's working.  So they get stripped of their positions as soon
as possible after the info has been transferred to the doc string.  For
example, SWPs cannot be dumped by the portable dumper.

> - My gut tells me that changing backquote can't be right.

I tend to agree.  I put the code into backquote-process when having
problems with things like:

       (mapatoms #'(lambda (,(car spec)) ,@body)

in cl-macs.el, where it's impossible to know where the doc string (if
any) is until after the expansion of the backquotes, or even at run time
(as here).  In the latter case, rather than "posifying" the doc string
at macro expansion time, we have to generate code to do it at run time.

>   (lambda (f) ..) *can* appear within a backquote without it being an
>   actual lambda expression.
>   What alternatives have you considered?

Not a lot of them, as yet.  Maybe testing for (function (lambda ...))
would be safer.  But I think I should try and find some other way of
solving these problems than altering backquote.el, which should be
easier now that the code is basically working.

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).