bug#67455: (Record source position, etc., in doc strings, and use this in *Help* and backtraces.)

[Stefan Monnier]

> We now have two distinct uses of SWPs: providing warning source locations
> to the compiler (where we want to keep the position as long as possible)
> and providing position information for the doc string (where we want to
> strip the position from the symbol ASAP, to avoid trying to use the SWP
> when we need a plain symbol).  If both of these occur together, we want
> to keep the SWP.

I think I'm beginning to understand.  So in the "load from source case",
some of your symbols are SWPs and you want to turn them into bare
symbols "on the fly" during macro-expansion rather than via a separate
"strip" phase, so you want the macro expansion to know whether it's done
for "load from source" or for some other purpose and you use
`byte-compile-in-progress` as a proxy for that information.
Is that it?

If so, (and if the stripping happens within macros), then indeed passing
it as a separate argument through all the recursive calls to
`macroexp--expand-all` would be cumbersome.  But I suggest you use
another name for that(e.g. `macroexp-strip-position`) so the
intention is made more clear.

Better yet: to avoid the problem of dynamic scope extending "too far"
(i.e. accidentally applying to nested loads/evals/compile/...), you
could put the relevant info into `macroexpand-all-environment`.
[ That var is also dynamically bound, but we're already careful to
  rebind it when needed so it doesn't apply to nested uses
  of macroexpansion.  ]

>> > By "currently", I mean that a defining form such as defun or defvar has
>> > commenced, but not yet terminated; its functions currently occupy stack
>> > frames.
>> So you mean we're inside `Fdefalias` or `Fdefvar_1`?
> Yes, or inside a macro (defun, defmacro, ...) which expands to a
> defalias.

These are *very* different times: `Fdefalias` and `Fdefvar_1` are
executed long after macroexpansion.  And they're small C functions which
run almost no external code at all, so they "occupy stack frames" only
for a very short time.

My crystal ball suggests that "currently" may be the wrong way to think
about it: maybe instead of thinking of "when" (as in "during the
definition of function FOO") what you're looking for might be "where"
(as in "within the body of FOO").
[ That's the same difference as the difference between dynamic and
  static scoping.  ]

If my crystal ball is right, then the better place to put that info is
probably `macroexpand-all-environment`.

> Ideally, I would like to have bound defining-symbol inside defun.

    (defmacro my-defun (name args &rest body)
      `(cl-macrolet ((defining-symbol () '',name))
        (defun ,name ,args ,@body)))

    (my-defun my-foo (x) (list x (defining-symbol)))

    (symbol-function 'my-foo)
    ==> #f(lambda (x) [t] (list x 'my-foo))

`cl-macrolet` uses `macroexpand-all-environment` for that.

> Fload uses read-positioning-DEFINED-symbols, as contrasted with the
> compiler, which uses read-positioning-symbols.  r-p-d-s positions only
> lambdas and NAMEs.  r-p-s positions all symbols except nil.

I think I'm beginning to understand (I guess I was struggling with your
use of "position" as a verb for some reason which made me think that
symbols were being moved rather than gaining position information).

So, IIUC you use `read-positioning-DEFINED-symbols` instead of
`read-positioning-symbols` because it's cheaper?  Do you have rough
numbers comparing the cost of `read`, `read-positioning-symbols`, and
`read-positioning-DEFINED-symbols`?

Also, IIUC you don't have a separate phase to strip the SWPs when
loading from source, but instead you strip them as you "consume" their
info during macroexpansion.  If so, how/when/where do you strip the
false positives that may occur inside quoted data or in code like:

    (defmacro foo (lambda bar) ...)
    (defmacro foo (defun bar) ...)

    (let* ((lambda foo)
           (defun bar))
      ...)

> Ah, right.  I hadn't considered this before.  The changes are by their
> very nature essentially complicated and difficult to understand.

[ Hmm... maybe not the best salespitch.  ]

>> I think you can simply wait to add the entry to
>> `macro-declarations-alist` until a later time, so the `defining-symbol`
>> thingies will be ignored during the early bootstrap and once we have
>> more infrastructure in place we can then register the handler on
>> `macro-declarations-alist`.
>
> This will not be simpler.  It would involve re-evaluating defun, then
> compensating for all the functions up to now whose NAMEs had been read
> without positions.

Not at all.  Those will remain without position, but only in
`src/bootstrap-emacs`.  In the real `src/emacs` they will get the
position because they'll come from the `.el[cn]` file and by the time we
get compile those files `macro-declarations-alist` will be
fully populated.

> There is unavoidable conplexity, here.

I'm definitely not convinced.  I suspect you've been asking yourself
"can it be made simpler" and you may indeed then convince yourself that
the answer is no, because of assumptions you don't reconsider.

Try instead to think about "what would it take to remove that complexity?".

> I see things somewhat differently.  We shouldn't increase the debugging
> burden even on "expert users".

Yet it's imposing more complexity (and hence more debugging burden) on
those same expert users.  🙁


        Stefan