bug#59140: 29.0.50; iter-yield from lambda

[Stefan Monnier]

> Notice, that the above code does not visually appear to use any lambdas,
> but it fails, because internally it expands to lambdas, causing
> (void-function cps-internal-yield).
>
> Now, let's get back to the original problem - the fact that in "local
> CPS" the "locality" of the lambda must be completely known. If we look
> at `named-let` it appears to be completely self-contained: the control
> flow and the usage of higher order functions that it expands to is fully
> known due to the way it is encoded into the macro.

For `named-let` we don't have such a guarantee.  You can do:

    (named-let my-foo ((x 1) (y 2))
      [...]
      (mapcar #'my-foo ...)
      [...]
      (set-process-filter proc #'my-foo)
      [...])

In most use cases, tho, we do, indeed.

> So provided with this information, do we have enough information to
> perform a CPS conversion of the `named-let`?

Usually, there is enough info to do it locally, yes.
[ We don't currently analyze enough of the code to have that
  information, OTOH.  ]

> Similarly, in pcase, unless the handler is provided as a lambda by the
> user, the internal/expanded lambdas might be "local enough" to perform
> CPS conversion.

For `pcase`, yes, definitely (and indeed, we'd want to compile those
lambdas into something more efficient).  It's more: we know they're
always called in a "tail call" position, so they can (in theory) be
treated by the CPS as continuations rather than arbitrary functions.

> What I am trying to say is that, if "local CPS conversion" of higher
> order functions is impossible to perform in general, then could we
> possibly make conversions on a macro per macro basis, when the macro is
> "local enough"?

Yes, that would be very welcome.  The easiest might be to make
`generator.el` operate on the code before macro-expansion (so it
gets to see `pcase` and `named-let` and can dispatch to ad-hoc code like
it does for special forms).  Another option is to improve `generator.el`
to the point where it can recognize the kind of code generated by macros
like `pcase` and `named-let` loops and treat it correctly.
Yet another option might be to arrange for macros like `named-let` and
`pcase` to leave hints in their code that make it easier for
`generator.el` to do its work.


        Stefan