Re: Gap buffer problem?

[Gerd Möllmann]

Pip Cet <pipcet@protonmail.com> writes:

> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>
>> Pip Cet <pipcet@protonmail.com> writes:
>>
>>> Gerd Möllmann <gerd.moellmann@gmail.com> writes:
>>>
>>>> Pip Cet <pipcet@protonmail.com> writes:
>>>>> if we ever replace the gap buffer code, we should make sure its
>>>>> replacement actually handles buffer text and text properties/intervals
>>>>> in an integrated manner, rather than storing just buffer text).
>>>>>
>>>>> Pip
>>>>
>>>> And if I may add a wish to the future author: Make whatever you use
>>>> persistent data structures, so that one could think of letting redisplay
>>>> run concurrently. Really! :-)
>>>
>>> You won't be surprised to hear I've been playing with some code,
>>
>> Indeed, I was just thinking to myself "I knew it" :-).
>> Two thumbs up!
>>
>>> so could I ask you to expand on this point? What precisely does
>>> redisplay require? Full snapshotting or would it be sufficient to have
>>> fine-grained locking?
>>
>> Maybe it's helpful when I tell something about the background. Some time
>> last year I asked myself if I could make Emacs more than one of my
>> plenty of CPU cores without solving the multi-threaded Elisp problem.
>> And the idea was that I could do that, possibly, by letting redisplay
>> happen in another thread.
>
> This may be a very stupid idea, but why not use a separate process?

Not stupid at all. I thought about something similar in a different
context, namely if one could decouple the GUI part of Emacs from the
rest.

Something like that has been done by Eberhard Mattes for OS/2 with the
old redisplay. He had to do that because the whole Presentation Manager
(GUI) in OS/2 would block, for all process, when an application did not
timely handle events and return to the PM. Something like that.
Eberhard's OS/2 Emacs had one process doing the GUI stuff, and one for
the rest. Both communicated with each other using a defined message
protocol. It worked. Don't remember what he used for process
communication, pipes or something else.

I got stuck with this idea because everything seemed to depend on
everything else nowadays. Redisplay needs to execute Lisp, Font backends
I think, not sure. Some GUIs call redisplay (nsterm). And then I
imagined the licensing issues, and dropped the idea. Although - NS could
really need something done, IMO, which was the reason I thought about
that in the first place. NS is not working for me at least. I always
wonder why nobody else has the same freezing problems that I have.

I think the same dependency problems also creep up with to concurrent
redisplay, don't know. Values of variables, faces, jit-lock, and so on.
I think it would be "easier" to handle if one has everything in one
process.

But in principle both could be done. An actor model.

> fork() is fast on GNU/Linux, and I suspect on macOS too, and the
> redisplay child would receive a consistent snapshot of the data to
> inspect and/or modify while coming up with the redisplay instructions,
> which it would then send back via a pipe or shared memory to be executed
> in the main process.
>
> I suggested doing something similar for GC (the GC child would perform a
> full GC and send back the Lisp_Objects which are definitely unreachable
> via a pipe. No, I never figured out how to make that work for weak hash
> tables which may resurrect references, I just made all hash tables
> strong...), and in that case the pipe seemed sufficient for the amount
> of data that was transferred, but I'm not sure how compact (or
> otherwise) serialized redisplay "instructions" would be.
>
> One issue I see is that fork() does a lot of housekeeping work in
> addition to marking the child's memory as a COW copy of the parent's
> memory at the time of the fork(). ISTR you can split that process on
> GNU/Linux (probably not Android), so you'd already have a prepared
> thread/LWP which wouldn't need to "start up" when you un-share the
> memory, but I can't find the relevant manpage right now. However, I have
> no real idea just how bad the fork() latency would be (as you point out,
> most people have more CPU cores than they can use, so I don't consider
> the approximate doubling of CPU usage a problem).
>
> This would deal very nicely with fontification code attempting to modify
> data it shouldn't, by ignoring such modifications. It would also deal
> with catastrophic failure in the redisplay code, as it's insulated in a
> separate process and we could just print a nice message in the main
> process rather than crashing all of Emacs.
>
> I'm emphatically not suggesting letting the redisplay child actually
> communicate with the X server or equivalent. That would be much more
> difficult.
>
> In fact, I think a good way to test this approach would be to use the
> tty code, since there's already a standard serialization of redisplay
> instructions for tty displays: VT100 escape sequences.
>
>> I later realized while thinking about the details, that this undertaking
>> is an order of magnitude too large for me. Everything taking more than a
>> few months is. And, in addition, I wouldn't want to do data structures
>> in C anyway.
>
> I think the VT100 case could be done as a weekend project (those always
> end up taking several weeks for me...), but I'm not sure it's worth it
> as VT100 redisplay isn't the common use case, and the performance
> problems are more visible on GUI terminals.

Yes. In a way, it's already the case that the GUI part of Emacs that I
described above for OS/2, is the terminal emulator, and the protocol is
VT100.

> And, like pretty much all Emacs ideas, this depends on having a better
> GC.
>
> (However, I've just experimented with an 8 GB process forking, and it's
> much slower than I'd hoped for - about 70 ms.  I wouldn't be surprised
> if most of that cost is setting up page tables for the ridiculously
> small 4KB page size x86 uses, so it may work a lot better for AArch64
> systems such as yours).
>
> Pip