Expansion of macros at compile time in eval-and-compile bodies (was: Re: [native-comp] Small, new bug with macros?)

[Adam Porter]

Hi Andrea,

Andrea Corallo <address@hidden> writes:

> Hi Adam,
>
> I've looked into the issue.  I think the code in bufler.el generting
> the error reduces to something like:
>
> #+BEGIN_SRC elisp
> ;;; bufler.el  -*- lexical-binding: t; -*-
>
> (defmacro bufler-defauto-group ())
>
> (eval-and-compile
>   (if 1
>       (bufler-defauto-group)
>     (bufler-defauto-group)))
> #+BEGIN_END
>
> I think the compiler error is correct because `bufler-defauto-group'
> is not defined in the compilation environment.  You should get the
> same error from a fresh Emacs instance just byte-compiling.

Yes, you seem to be right about that.  If I run, on that file:

  emacs --batch --file bufler.el -f emacs-lisp-byte-compile

I get the same error:

  Symbol’s function definition is void: bufler-defauto-group

> The fix should be to wrap its definition within `eval-and-compile'
> too.

Yes, that seems to fix it.  But I don't understand why.

There must be something I'm misunderstanding about how loading and
compilation work.  In the Emacs 26.3 Elisp manual, in section 14.3, it
says:

  In order for compilation of macro calls to work, the macros must
  already be defined in Lisp when the calls to them are compiled.  The
  compiler has a special feature to help you do this: if a file being
  compiled contains a ‘defmacro’ form, the macro is defined temporarily
  for the rest of the compilation of that file.

That would seem to imply that the error shouldn't happen, because the
macro is defined in the same file before it is called.

The next paragraph says:

  Byte-compiling a file also executes any ‘require’ calls at top-level
  in the file, so you can ensure that necessary macro definitions are
  available during compilation by requiring the files that define them

Then in section 17.5 of that manual, in the listing for
eval-and-compile, it says:

  This form marks BODY to be evaluated both when you compile the
  containing code and when you run it (whether compiled or not).

  You can get a similar result by putting BODY in a separate file and
  referring to that file with ‘require’.  That method is preferable
  when BODY is large.  Effectively ‘require’ is automatically
  ‘eval-and-compile’, the package is loaded both when compiling and
  executing.

Does that mean that the body of an eval-and-compile form is evaluated in
a different environment than the one defined by the file containing the
eval-and-compile form?  If so, putting the macro definition in an
eval-and-compile form wouldn't seem to imply that a macro defined in it
would be available in subsequent eval-and-compile forms.  Does that mean
there's some kind of parallel environment for bodies of any
eval-and-compile forms in a file?

Maybe I'm just missing something obvious, but I wonder if the manual is
missing an explanation of this finer point.  It does say, "Most uses of
‘eval-and-compile’ are fairly sophisticated."

> Because the native compiler spawn sub-processes for async compilations
> it is a little more severe into highlight this tricky issues into the
> code.

Maybe so.  But as long as it has the same behavior as the byte-compiler,
that seems correct, right?

> Changing topic I have no idea why this instead go through.  To me looks
> like a bug, isn't?
>
> #+BEGIN_SRC elisp
> ;;; bufler.el  -*- lexical-binding: t; -*-
>
> (defmacro bufler-defauto-group ())
>
>  (eval-and-compile
>    (bufler-defauto-group))
> #+BEGIN_END

You mean that byte-compiling that file doesn't produce an error, right?
And, yes, you're right, it doesn't.  That confuses me as well.  I don't
understand what's going on here.

In case this does indicate a need to improve the manual or fix a minor
bug, I'm moving this to a new thread.

I have a sense of déjà vu about this; I think a similar issue came up
several weeks ago, which I also asked you about.  So thanks for your
help here, and my apologies if I'm using your time on an issue that
isn't your problem.