Re: Is there any easy way to fork in elisp? Why not?

[Philipp]

> Am 26.10.2021 um 14:44 schrieb Rudi C <rudiwillalwaysloveyou@gmail.com>:
> 
> There are some scripts that I find convenient to code in elisp,  and I like 
> to be able to run these with little startup time. 
> 
> The natural solution is to use `emacs --batch --eval` together with 
> `server-eval-at` to reuse an already loaded server, but this solution needs a 
> lot of throwaway servers already started and a load balancer. (Is there a 
> Jupyter kernel for elisp? I already have a prototype server 
> (https://github.com/NightMachinary/JupyterGarden) that starts throwaway 
> Jupyter kernels and load-balances requests to them.)
> 
> If the server could just fork, eval the request, return it, and die, 
> everything would be easy and painless. 
> 
> The fork API could be:
> ```
> (let-fork ((some_val (expensive_compute)) ...)  
>  (lambda()
>    (message "Computed: %s" some_val)))
> ``` 
> Where emacs forks for every binding, and computes the results, and finally, 
> asynchronously calls the supplied callback function with all of the bindings 
> available, similar to a closure.
> 
> I am sure there are better API designs, but my point is that this fork API 
> should be very easy and work with synchronous third-party code, just like a 
> shell (or GNU Parallel). Parallelism via multiprocessing should be easy and 
> painless.
> 
> The third-party packages for parallelism that I have skimmed, start a fresh 
> emacs instance, and do not fork, which is useless, as all the needed data and 
> behavior needs to be manually reloaded by the user.

If you're talking about the Unix fork system call (or similar calls such as 
clone), then the answer is that it's not possible to fork and do interesting 
work in the child from any process that might use multiple threads (which 
includes most "interesting" interactive programs, and definitely includes 
Emacs). The reason is that the other threads might be in a critical section, 
but since the threads are gone in the child the critical section can never 
left, so the child has to avoid any code that might enter any critical section, 
which precludes calling any function that isn't async-signal-safe, and even 
heap memory allocation isn't async-signal-safe. Therefore programs can 
generally only call execve (which removes the critical sections altogether) 
plus a small number of other functions to set up the child process (chdir, 
setsid...) after forking.

If you're talking about some more generic concept similar to the Unix fork 
function implemented completely in userspace (e.g. by creating a thread and 
either eagerly or lazily copying the entire state of the Lisp interpreter), 
then such a thing would be possible in principle, but face quite some 
difficulties. The copying is difficult to get right, and for some objects (such 
as processes) it's hard to define what the semantics of the copy should be. 
It's much easier to start with a minimal known-good interpreter state and 
modify only the pieces of it which are needed to execute the code in question, 
which is exactly what the asynchronous frameworks such as emacs-async are doing.

Another relatively common way to achieve parallelism is to share VM between 
multiple processes ("threads"). However, that requires carefully synchronizing 
every access to shared mutable state, and Emacs Lisp code generally doesn't do 
that, so this approach is infeasible for Emacs as well.