Re: continuation passing in Emacs vs. JUST-THIS-ONE

[Tomas Hlavaty]

On Thu 16 Mar 2023 at 23:08, Stefan Monnier <monnier@iro.umontreal.ca> wrote:
>>> ;; (futur-let*
>>> ;;   (exitcode <- (futur-process-make :command cmd :buffer t))
>>> ;;   (out (buffer-string)) ;; Get the process's output.
>>> ;;   (cmd2 (build-second-arg-list exitcode out))
>>> ;;   (otherexit <- (futur-process-make :command cmd2 :buffer t)))
>>> ;;  (futur-pure (buffer-string)))
>>
>> Seems like beautiful lisp code has no futur. :-)
>
> BTW the above code can't work right now.

That is a shame.

> Part of the issue is the
> management of `current-buffer`: should the composition of futures with
> `futur-let*` save&restore `current-buffer` to mimic more closely the
> behavior one would get with plain old sequential execution?  If so,
> should we do the same with `point`?  What about other such state?

I do not think there is a good implicit solution.
Either it would save too much state or too little,
or save it the wrong way.

It should be written out explicitly (like proc-writer below, for
example).

> The `futur-progn` is just:
>
>     (defmacro futur-progn (form &rest forms)
>       (if (null forms) form
>         `(futur-let* ((_ ,form)) (futur-progn ,@forms))))

Nice, this is much better.

> As for passing the result of `futur-let` to `:command` it just requires
> writing `futur-process-make` in a way that is tolerant of this
> `:command` arg being a future rather than a string, which should be
> fairly easy (it's basically always easy when done within a function
> which itself returns a future).

Sounds good.

>> or would it need some fancy syntax rewriting like other async/cps
>> syntax rewriting libraries?
>
> I don't think so, no.  But you would need fancy rewriting if you wanted
> to allow
>
>     (concat foo (futur-let* (...) ...))
>
> But as you point out at the beginning, as a general rule, if you want to
> avoid rewritings in the style of `generator.el`, then the code will tend
> to feel less like a tree and more "linear/imperative/sequential",
> because you fundamentally have to compose your operations "manually"
> with a monadic "bind" operation that forces you to *name* the
> intermediate value.

That's what I suspected.

Being forced to name the values leads to very bad code.

>> Second question: I see that futur-wait blocks the whole Emacs due to
>> the while loop.  How can one use futur without blocking Emacs?
>
> Don't use `futur-wait` and instead use `futur-let*`.
> IOW: instead of waiting, return immediately a future.

Understand, thanks for clarification.

>> Last question: How would similar functionality be implemented
>> using futur?
>
> Good question.
> To a large extent I guess it could be implemented in basically the same
> way: you'd use futures only for the timer part of the code, and leave
> the process's output to fill the buffer just like you do.
>
> I think the difference would be very small and cosmetic like replacing
>
>     (defun stream-pull-in-background (stream &optional secs repeat)
>       (let (timer)
>         (setq timer (run-with-timer
>                      (or secs 1)
>                      (or repeat 1)
>                      (lambda ()
>                        ;;(message "@@@ polling!")
>                        (unless (funcall stream)
>                          (cancel-timer timer)))))))
>
> with something like:
>
>     (defun stream-pull-in-background (stream &optional secs repeat)
>       (futur-run-with-timer
>        (or secs 1)
>        (lambda ()
>          ;;(message "@@@ polling!")
>          (when (and (funcall stream) repeat)
>            (stream-pull-in-background stream secs repeat)))))
>
> The only benefit I could see is that it returns a future, i.e. a kind of
> standardized representation of that async computation so the caller can
> use things like `futur-wait` or `futur-let*` without having to care
> about whether the function is using timers or something else.
> And, there's also the benefit of standardized error-signaling.

Given that there is no working example and state management is not
really thought through, it is hard to imagine, what do you mean exactly.

I do not want to block emacs.
futur-wait blocks emacs.

I also do not understand, why would you use a timer and poll.
The functionality is edge triggered push model where this does not
make sense.

Here is the edge triggered push model example written in the inverse
style of the level triggered pull model example using streams I sent
earlier:

(defun message-writer ()
  (lambda (string)
    (when string
      (insert (format "%d %s\n" (length string) string)))))

(defun proc-writer (buffer writer)
  (lambda (string)
    (when string
      (with-current-buffer buffer
        (let ((proc (get-buffer-process buffer)))
          (if proc
              (let* ((mark (process-mark proc))
                     (moving (= (point) mark)))
                (save-excursion
                  (goto-char mark)
                  (funcall writer string)
                  (set-marker mark (point)))
                (when moving
                  (goto-char mark)))
            (save-excursion
              (goto-char (point-max))
              (funcall writer string))))))))

(defun line-writer (writer)
  (let (line)
    (lambda (string)
      (if string
          (let ((x (split-string (concat (or line "") string) "\n")))
            (while (cdr x)
              (funcall writer (pop x)))
            (setq line (car x)))
        (when (and line (not (equal "" line)))
          (funcall writer line)
          (setq line nil))))))

(defun writer-filter (writer)
  (lambda (_proc string)
    (funcall writer string)))

(defun writer-sentinel (writer)
  (lambda (proc _event)
    (unless (process-live-p proc)
      (funcall writer nil))))

(defun writer-process (buffer-name command writer)
  (let* ((b (test-buffer buffer-name))
         (w (line-writer (proc-writer b writer))))
    (make-process :name buffer-name
                  :command command
                  :buffer b
                  :sentinel (writer-sentinel w)
                  :filter (writer-filter w))))

(defun test4 (buffer-name command)
  (writer-process buffer-name command (message-writer)))

;;(test4 "test4" '("cat" "/tmp/a.el"))
;;(test4 "test4" '("find" "/home/tomas/mr/" "-type" "f" "-name" "*.el"))

In the edge triggered push style, it is important to not miss any
events, which is what writer-filter and writer-sentinel do (and one can
also see that in your comment FIXME: If the process's sentinel signals
an error, it won't run us; note that in the pull model, this issue does
not exist and the code is much simpler).  line-writer splits input into
lines.  proc-writer manages the external state explicitly and
message-writer is the actual functionality I want to achieve in the
example (output lines and their length).  writer-process is a generic
driver to run a command in background and do something per each line of
output, push style.

The advantage of this push model is that it does not require "infinite"
buffer and the computation happens as soon as possible without polling.

The disadvantages are numerous.  For example, the pace is dictated by
the outside process which can overwhelm emacs and make it unuseable.
Another serious problem is that doing too much in the filter function
can make emacs unuseable (iirc that was one of the issues the original
poster in this thread complained about).  Even more serious problem is,
that C-g in filter function does not work and leads to abort (maybe that
is the reason C-g is not very reliable and I need to use more than one
emacs process).

In futur.el, you do not use filter function but it seems that futur.el
combines the worst features of both pull and push models, e.g. event
detection, infinite buffer, polling and even blocking emacs.
Why do you recommend to poll with futur.el?