Re: Circular records: how do I best handle them? (The new correct warning position branch now bootstraps in native compilation!)

[Alan Mackenzie]

Hello, Stefan.

On Thu, Dec 30, 2021 at 13:37:47 -0500, Stefan Monnier wrote:
> >> >> Hmm... circularity is quite normal in data structures, yes.
> >> >> But presumably this is only applied to source code where circularity is
> >> >> very rare.  Could it be that you end up recursing in elements which
> >> >> actually aren't part of the source code (and hence can't have
> >> >> symbols-with-positions)?
> >> > I honestly don't know at the moment.
> >> I think it's worth the effort to try and track this down.  Maybe we can
> >> completely circumvent the problem.
> > I don't think there are any such cases.

> Hmm... I thought this whole circular records thread started because you
> bumped into such a case.  I feel like I'm misunderstanding something.

What I bumped into was circularly linked vectors in the source code
being compiled.

I've amended the reader so that it doesn't put positions on symbols
which are read as components of other structures such as byte compiled
functions, text property lists in strings, and so on.  (Actually, there
was very little to amend.).

I've amended macroexp-strip-symbol-positions so that it ignores
circularity unless it hits an infinite recursion, in which case it
starts again, recording all components found in hash tables.

I committed these changes a short time ago.

> >> So the symposes can end up in 2 places:
> >> - in the .elc file: no need to strip the pos here, just make sure the
> >>   symbols get printed without their position.
> > The positions get stripped before the code is dumped to the .elc.

> Why bother?  You can just have a `print-symbols-without-position` which
> you let-bind around the printing code.

I think I've got that already, though it's a long time since I looked at
it.

> >> - elsewhere: that's the problematic part because this only occurs where
> >>   the source code gets stealthy passed elsewhere, e.g. when a macro
> >>   calls (put ARG1 'foo ARG2) during the macro expansion (rather than
> >>   returning that chunk of code in the expansion).
> > This isn't a problem.  If it is a compiled macro doing this, the
> > positions will already be gone from the symbols.  If it is from an
> > uncompiled macro, XSYMBOL in Feval's subroutines does the Right Thing.

> I didn't mean sympos coming from the macro but sympos coming from the
> args passed to the macro.  Something like:

>     (defmacro foobar-really (arg1 arg2)
>       (puthash arg1 arg2 foobar-remember)
>       `(progn (do-something ,arg1) (do-something-else ,arg2)))

Args which are symbols with positions are first and foremost symbols.
They behave like symbols when used.  The test of this is that Emacs
bootstraps, despite having many macro expressions like ,@body which
expand to expressions with positions.

> The `remember` property will end up containing symbols-with-pos if
> `arg2` contains symbols.

In that (puthash arg1 arg2 foobar-remember), if the key is a symbol with
position, it is stripped.  I think the value will keep its positions.
This might still be a problem.

> > It's used all over the place.  In eval-when/and-compile, it is used
> > before the evaluation.  It is used before dumping the byte compiled code
> > to the file.elc, and before passing this code to the native compiler.
> > Several (?most) of the byte-compile-file-form-... functions use it.
> > It's used in the newish keymap functions near the end of bytecomp.el, in
> > byte-compile-annotate-call-tree, etc.  Also in cl-define-compiler-macro,
> > and internal-macro-expand-for-load.

> Interesting.  Why do you need it at so many places?
> What is it usually used for?

Stipping positions from compiled code before dumping it to an .elc file,
and also before passing the compiled code to the native compiler.

The fact is these positions on the symbols are unwanted for most uses of
symbols around compilation, being needed only in the analysis phase of
the source code.

> > Additionally, also from Fput, to prevent symbols with positions
> > getting into symbol property lists.

> IIUC this is for the kind of example I showed above (tho I used
> `puthash` instead of `put`)?

> > Yes, I have to do this.  I am still debating whether just to do it
> > (which might slow things down quite a bit), or to do it in a
> > condition-case handler after the recursion has exceeded the 1,600
> > max-lisp-eval-depth.  I'm inclined towards the latter at the moment.

> Using a (weak) hash-table may actually speed things up if you call it
> from lots and lots of places and it thus ends up being applied several
> times (redundantly) to the same data.

In the end I went with the condition-case approach, based on a gut
feeling that hash tables aren't very fast, and they're needed only
rarely, certainly whilst bootstrapping Emacs.

> > For other Lisp objects with a read syntax, such as char tables and
> > decorated strings, I intend to amend the reader just to output plain
> > symbols for them.

> Sounds reasonable.

I've done this now.  Feel free to look at the new version of
scratch/correct-warning-pos.

And a Happy New Year!

>         Stefan

-- 
Alan Mackenzie (Nuremberg, Germany).