Re: Unicode support

[Kenichi Handa]

Sorry for not joining this important topic much earlier.

Eli Zaretskii <address@hidden> writes:
> In other words, the main connection between Lisp and display is indirect: 
> Lisp programs change the buffer text or variables which affect how the 
> text is displayed, and the next redisplay reflects those changes.

But, we have font-lock (jit-lock), and a Lisp program is
called while redisplaying.

>>  I don't know which language would be suited best.  Probably as with
>>  the rest of Emacs, some core C functions should be accompanied with
>>  Lisp code (probably mostly in CCL).

> I think pretty much everything will have to be in C, since redisplay is a 
> very complex piece of code and its performance is critical for Emacs 
> responsiveness.  Even today, if you disable most of the special 
> optimization cases in redisplay, it becomes unbearably slow on anything 
> slower than a 500-MHz machine, even for such trivial operations as cursor 
> motion.  So any code added to redisplay should run as fast as possible.

I think jit-lock/font-lock is very fast even if we run Emacs
Lisp code every time a visible part of a buffer is changed.

And, what we have to do for such complicated script as
Indic, Arabic, etc is just one thing: put proper
compostition property on a text on the fly.  And, that task
is, I believe, relatively faster than what the current
font-lock is doing.

>>    . For Arabic scripts, adding initial, medial, final, and isolate
>>      properties to the characters.

I and Takahashi-san are now locally doing an experiment of
compostion-lock-mode to support various presentation forms
of Arabic on the fly.  The buffer contents are
U+0600..U+06FF, but we put characters of U+FB50.. (Arabic
Presentation Form) as a composition property of each buffer
character by Lisp programs called from display engine.  This
is the same way as font-lock putting `face' property to a
text.

> Shouldn't Arabic presentation forms addition be delayed for some (short) 
> time, in case the user types more characters of the same word?  Emacs 
> will not normally know how to display the character just typed: as 
> initial, medial, or final form, since it doesn't know what will be typed 
> next.

Our current code produces no delay.  At least, it is fast
enough so that a user can type arabic text without any
flustration.  When I type SPC within a Arabic word, the
provious char changes shape to final form, and the following
char to initial form instantly.

>>    . For Indic scripts, reordering of glyphs according to the Unicode
>>      standard.

Kawabata-san is now working on it.  But, I haven't heard
from him about a progress for a month.

>>    . For Right-To-Left scripts, reeordering from logical to visual
>>      order, taking care of nested levels etc. as described in the
>>      Unicode standard.

Yah!  This is the most difficult thing, especially the
nesting is.

>>    . For everything else, applying data from the various Unicode tables
>>      to make the glyph engine aware of non-spacing accents, combining
>>      characters, etc.

> We discussed similar issues related to bidirectional support with Gerd.  
> The current state of thinking, as I understand it, is that the 
> rearrangement of characters for display (i.e. conversion between the 
> logical and visual order) should happen inside the functions which walk 
> the buffer and supply the next character for display to the next layer 
> (which generates glyphs).

And, I'm not yet convinced that such a way is better than
simply prefetching display elements and reordering them in
advance.  :-)

---
Ken'ichi HANDA
address@hidden