Re: Elisp containers

[Vladimir Sedach]

> I think the basic idea is to avoid data races while also avoiding the
> overhead of a separate process.  So the design should keep things in
> process (duh) and also reject shared mutable state; instead objects must
> be explicitly sent between threads.  (If you lift this latter
> requirement then you really just have what we have now.)

This problem boils down to "what is the right way to virtualize a
single address space Lisp system?"

> So, first, what environment should a new thread inherit?
>
> If it is a clean environment -- say a copy of the dumped state -- then
> that means that user changes won't affect threads.  And, threading code
> will need to know exactly what things must be copied into the new
> container.
>
> This seemed unfortunate to me, not to mention hard on thread users,
> because it's difficult in an open environment to know what may be
> required.  (You can see echoes of this problem in the mhtml mode, where
> it has to use a regexp on symbol names, eww, to capture locals.)

> However, if it is not a clean environment, then thread creation will be
> very expensive.  You will have to copy much of the heap into the new
> thread.  (Perhaps the copying could be done lazily at the expense of
> slowing down access to global variables.)

One way to solve the problems of "which environment to start from?"
and expensive copying is to implement environments as a fully
persistent (AKA purely functional) table. This will let you do
roughly the equivalent of fork(2) in a single address space quickly
(with the real cost amortized over the whole system). You can also
use this for parent/child fork parallelism where the children inherit
the parent's dynamic bindings.

Having single address space share-nothing threads IMO defeats the
main benefits of a single address space, which are: eliminating data
duplication, and zero copy message passing. It is also a lot more
work to implement than using multiple Emacs OS processes.
Virtualizing the environment with a shared heap is an IMO interesting
idea that has not been well explored yet, but it requires a very
different approach to Lisp system functions (they will all have to be
based around immutable data structures), and more to the point is not
relevant to "Elisp containers."

> One idea I had to reduce the cost here was to have a special case: when
> a thread dies, let thread-join simply transfer the data from that thread
> to its own heap.

Cilk has a very well thought out model of lightweight thread
forking/joining using cactus stacks (an idea derived from Interlisp's
spaghetti stack), where child threads return their data to the parent.

> Finally, this approach greatly reduces debuggability and the ability to
> mess around.  Emacs would no longer be an open system -- threads would
> have private data and there would be no way to access that.

The debugger could point to the environment of the thread from which
it was invoked. The same could be done for REPLs. So the whole
tool set would have to be changed to accommodate this virtualization.

Multiple Emacs processes have the same problem, but there are already
working solutions that can be adapted for async.el. Something like
the SWANK protocol can be used to provide debugging and a REPL for
multiple Emacs processes, and not just on the same machine but
remotely as well. The nice thing is that the Emacs processes already
show up in the process list, and would get their own names in SWANK,
which would make managing them as easy as managing containers.

Vladimir