bug#65017: 29.1; Byte compiler interaction with cl-lib function objects, removes symbol-function

[Stefan Monnier]

> We can simplify your nice little test case to
>
> ------- first file -----------
> (require 'cl-macs)
> (defun zeta () (cl-flet () #'equal))
> ------- second file ---------
> (defun eta () (cl-flet () (funcall #'equal 12 34)))
> ------------------------------
>
> and indeed, the leak is in cl--labels-convert-cache which will contain
> `equal` as a symbol-with-pos after byte-compiling the first file, and this
> causes trouble in the second file.
>
> cl--labels-convert-cache contains
>
>   (#<symbol equal at 49> function #<symbol equal at 49>)
>
> and the function `eta` is consequently defined as
>
>   (closure (t) nil (progn (#<symbol equal at 49> 12 34)))
>
> where 49 is the position of `equal` in the first file.

Thanks Mattias.

AFAICT the problem is that OT1H `symbols-with-pos-enabled` is
non-nil while loading the second file, but OTOH positions aren't
stripped from the resulting code.

So in `cl--labels-convert` when we check

    (eq f (car cl--labels-convert-cache))

we get when `f` is just `equal` whereas the cache contains
the sympos, but that sympos is not stripped later on.

I don't know why `symbols-with-pos-enabled` is non-nil at that point (I
thought we only enabled it wile byte-compiling), but assuming there's
a good reason for it, I tried to work around the problem with the patch
below which is conceptually correct (indeed we should only return
(cdr cl--labels-convert-cache) only in the case where `f` is exactly
the very same object as (car cl--labels-convert-cache)).

Sadly, it doesn't seem to help for a reason that escapes me.
The *Messages* buffer says:

    For information about GNU Emacs and the GNU system, type C-h C-a.
    Compiling .../tmp/foo1.el...
    Self-rewrite #<symbol equal at 82> to #'#<symbol equal at 82> (t)
    Compiling .../tmp/foo1.el...done
    Wrote .../tmp/foo1.elc
    Self-rewrite equal to #'#<symbol equal at 82> (t)
    Self-rewrite #<symbol equal at 82> to #'#<symbol equal at 82> (t)

where the middle "self-rewrite" is the culprit.
Apparently

    (let ((symbols-with-pos-enabled nil))
      (eq f (car cl--labels-convert-cache)))

returned non-nil when `f` was the bare `equal` and the `car` returned
the sympos.  How can that be?  I thought `eq` should really be the
good old "pointer equality" when `symbols-with-pos-enabled` is nil.

What am I missing?


        Stefan


diff --git a/lisp/emacs-lisp/cl-macs.el b/lisp/emacs-lisp/cl-macs.el
index 0a3181561bd..bc71f565f3b 100644
--- a/lisp/emacs-lisp/cl-macs.el
+++ b/lisp/emacs-lisp/cl-macs.el
@@ -2037,7 +2039,12 @@
    ;; *after* handling `function', but we want to stop macroexpansion from
    ;; being applied infinitely, so we use a cache to return the exact `form'
    ;; being expanded even though we don't receive it.
-   ((eq f (car cl--labels-convert-cache)) (cdr cl--labels-convert-cache))
+   ((let ((symbols-with-pos-enabled nil))
+      (eq f (car cl--labels-convert-cache)))
+    (let ((print-symbols-bare nil))
+      (message "Self-rewrite %S to %S (%S)" f (cdr cl--labels-convert-cache)
+               symbols-with-pos-enabled))
+    (cdr cl--labels-convert-cache))
    (t
     (let* ((found (assq f macroexpand-all-environment))
            (replacement (and found