bug#67455: (Record source position, etc., in doc strings, and use this in *Help* and backtraces.)

[Stefan Monnier]

> Some symbols must not be stripped.  For example, in cl-generic.el L403
> we have:
>
>     (fun `(cl-function (lambda ,plain-args ,@body)))              
>
> ..  There the position on the lambda must be preserved until ME2 time
> when it becomes clear what the shape of the lambda is.  In particular,
> whether there is already a doc string in ,@body to amend, or we need to
> insert a new one.

I could see some reasons you *may* want to keep some info here, but it's
definitely not a "must" because the source position of those functions
should generally not point to `cl-generic.el:403` but to where
`cl-defmethod` was used.

Also, if you do want to preserve some info there (presumably with the
intent to combine it with the more important info that will be available
at ME2) it will need cooperation from `cl-generic.el` because, as far as
the semantic of Emacs Lisp is concerned, the above constructs a list
with a `lambda` symbol inside of it, with no guarantee that it will be
used as a function, and even if ever used as a function there's no
guarantee that this list will pass through the few places where we strip
SWPs, so keeping SWPs in there without some explicit request from
`cl-generic.el` would be a bug.

IOW, I don't think it's a good reason to rule out

    (strip-all-symbol-positions
     (macroexp--expand-all
      (read-positioning-symbols)))

BTW, AFAIK the above is conceptually what the byte-compiler does (except
it performs a few more transformations between `macroexp--expand-all`
and `strip-all-symbol-positions`).  Is it the case that `cl-defmethod`
generates a function whose source position (partly) points to
`generic.el:403` if `cl-generic.el` was interpreted but not if
it compiled?

>> >> >> Also, IIUC you don't have a separate phase to strip the SWPs when
>> >> >> loading from source, but instead you strip them as you "consume" their
>> >> >> info during macroexpansion.  If so, how/when/where do you strip the
>> >> >> false positives that may occur inside quoted data or in code like:
>
>> >> >>     (defmacro foo (lambda bar) ...)
>
> (defmacro foo (lambda bar)
>   `(cons ,lambda ,bar))
>
> exoands to
>
> (macro closure (t) (lambda bar) ";POS^^^A^A^A [foo *scratch* 158 nil]
> " (list 'cons lambda bar))

IIUC your reader will make the `lambda` formal argument into an SWP.
Where is that SWP stripped?

> so it is clear this case is getting handled OK.  I'm afraid I can't
> point out the exact place in the code at the moment where this is
> getting done.

I think it would be good to know, so as to be able to decide whether
it'll indeed always work right, or we just got lucky this time.

>> I don't actually know whether it will be better.  It just seems it could
>> lead to simpler code, with no change at all to the reader, for example.
> The exercise is intrinsically complicated.

Could you explain what you think makes it intrinsically complex?

> What you're suggesting is that the code to decide which SWPs to strip
> is going to be simpler than the enhancements to the reader.

As seen above, I suggest to leave the reader unchanged and to strip all
SWPs.  I'm pretty sure it would give comparable info to what you have
and it would be simpler (also, it would make it much less likely to
have discrepancies between the compiled case and the interpreted case).
My main worry with it would be performance.


        Stefan