bug#67455: (Record source position, etc., in doc strings, and use this in *Help* and backtraces.)

[Stefan Monnier]

> I think it is better to regard the byte compilation as the special case.
> Only in byte compilation do we want to preserve the SWPs on forms
> getting posified.
[...]
> Byte compilation is NOT calling loading from source.  We don't have a
> caller/callee relationship here.  We are doing posification either from

The "callee" I'm talking about is `load-source-file-function` (which is
instead another "caller", beside the byte compiler), it is the
code that tests `byte-compile-in-progress`.

Based on the above two elements, I suggest the name
`macroexp--inhibit-strip-sympos` (and set/pass/bind it as needed in the
compiler).

[ FWIW, the reason I associate this variable with
  `load-source-file-function` rather than with the compiler is because
  the macroexp code which tests this variable only strips the SWPs at
  a very few different spots, more specifically those very spots that
  are expected to get SWPs in the `load-source-file-function`.
  Different minds work differently, I guess. 🙂  ]

>> Also, I think as a general rule it's better for the caller to set
>> a callee variable that controls how the callee behaves, rather than for the
>> callee to check a caller variable to decide how to behave, because
>> it's normal for the caller to "know about" its callee (after all, it's
>> the caller which decides to call the callee), whereas it's not normal
>> for the callee to know about specific callers (it creates undesirable
>> dependencies).
> byte compilation or from somewhere else.  It is analogous to testing
> lexical binding.  Here, the variable is called lexical-binding; it is
> not named after a particular activity to be carried out differently for
> l-b and not l-b.

Indeed testing `lexical-binding` in macros (like we do at a few places)
sucks; it's an ugly hack.  We use it because that was the least bad
option we could come up with given the need to preserve
backward compatibility.
Here there's no such problem.

>> The earlier the better, in theory, but not at any cost.
> No, the earlier the better, full stop.

Please "full stop" being absolutist.  We're talking about opinions and
preferences here.

When hitting an error, I spend more time reading the code (and modifying
it) than looking at debug output, so to me the clarity of the code is
more important than whether a few lambdas get some addition positional
info, especially since I usually know full well where those lambdas
come from.

I understand it affects us differently, but the tradeoff is real.

>> Having to write all that code within the very restrictive sublanguage
>> available before subr.el and backquote.el is a cost I don't think
>> justifies it.
> The cost has already been paid, by me.

Code is not "fire and forget".

>> >> My crystal ball suggests that "currently" may be the wrong way to think
>> >> about it: maybe instead of thinking of "when" (as in "during the
>> >> definition of function FOO") what you're looking for might be "where"
>> >> (as in "within the body of FOO").
>> >> [ That's the same difference as the difference between dynamic and
>> >>   static scoping.  ]
>> > I'm having trouble understanding what you're saying, here.
>> Is it because you don't understand the difference between dynamic
>> scoping and static scoping, or because you don't see the relationship
>> with that and your notion of "currently being defined"?
> The latter, I think.  defining-symbol is entirely dynamically scoped.

We're still miscommunicating.  You're talking about how your code is
implemented, apparently, whereas I'm asking about what is the
intended behavior.

It's like I'm asking what the C spec says and you're answering me by
telling me how GCC works.

> I'm convinced it does.  Can you suggest a scenario where the
> defining-symbol mechanism (outlined above) might fail?

Without knowing what it is intended to do, the only thing we can say is
that it does what it does, so no indeed it won't fail to do what it
does, since that's what it does. 🙂

>> >> Do you have rough numbers comparing the cost of `read`,
>> >> `read-positioning-symbols`, and `read-positioning-DEFINED-symbols`?
>> > No, but they will be very close to eachother (and very cheap)
>> Then I think we should use `read-positioning-symbols`, which
>> requires fewer code changes.
> It won't.  Required would be Lisp code to determine whether a particular
> SWP needs to be stripped or not.  This is not going to be simple.  It is
> likely to be about as complicated as the existing enhancements to read0.

They'd all need to be stripped, AFAICT, so we'd do:

    (strip-all-symbol-positions
     (macroexp--expand-all
      (read-positioning-symbols)))

What would be hard about it?

>> >> Also, IIUC you don't have a separate phase to strip the SWPs when
>> >> loading from source, but instead you strip them as you "consume" their
>> >> info during macroexpansion.  If so, how/when/where do you strip the
>> >> false positives that may occur inside quoted data or in code like:
>
>> >>     (defmacro foo (lambda bar) ...)
>> >>     (defmacro foo (defun bar) ...)
>
>> >>     (let* ((lambda foo)
>> >>            (defun bar))
>> >>       ...)
>
>> > There's a pcase arm right at the end of macroexp--expand-all which strips
>> > SWPs of their positions.  Recursing through macroexp--all-forms will
>> > eventually hit this pcase arm for these lambdas.

Actually, now that I look at the code I only see:

            ((guard (and (not byte-compile-in-progress)
                         (symbol-with-pos-p form)))
             (bare-symbol form))

is that the "arm" you're talking about?  AFAICT this will handle only
those symbols which appear as Lisp expressions (IOW symbols which are
variable references), so it will strip the `bar` in the second example
but not the `bar` in my first exmple, nor the two `lambda`s, nor
those in

    '(lambda (defun bar))

>> But your current code in byte-run.el is a Bad Thing as well.
> What, precisely, do you find bad about it?  It may be possible to improve
> it without wholesale redesign.

A lot of it is hard to read because it is constrained to a restrictive
subset of ELisp.

>> I'm not suggesting to drop support for lambdas loaded from source.
>> I'm saying we don't need to support it for the first N files loaded into
>> `src/emacs-bootstrap`.
> You're suggesting dropping support for many source files, where that
> support is most needed.

"Most needed" according to which criteria?

> I'm not playing on words.  My point is that
> read-positioning-defined-symbols exists and works.  It is not a
> speculative "would be nice to have".  The work has already been done.

Code costs by merely existing.

> Why do you think this design change will be better than the existing
> design?

I don't actually know whether it will be better.  It just seems it could
lead to simpler code, with no change at all to the reader, for example.
I'm here asking what lead you to the current design, under the
assumption that the complexity you introduced was the result of
other experiments.

Am I to understand that the current code is basically your first attempt
at adding such functionality?


        Stefan