Re: [emacs-bidi] Mixed L2R and R2L paragraphs and horizontal scroll

[Eli Zaretskii]

> From: Stefan Monnier <address@hidden>
> Cc: address@hidden, address@hidden, address@hidden
> Date: Thu, 04 Feb 2010 17:13:29 -0500
> 
> >> This how it is done in Hebrew newspapers and in HTML, and this is how
> >> Hebrew readers are used to it (of course newspapers and HTML don't
> >> breaks words, but that is the only difference).
> > Please!  Newspapers don't have truncated and continued lines, they
> > have newlines between every two lines.  With newlines, the bidi
> > display will show exactly what you (and every other Hebrew reader)
> > expect.
> 
> I'm not sure whether newpapers really have such newlines (at least for
> paper newspapers, it's a philosophical question).  But I think Ehud's
> argument makes sense if you consider that currently (i.e. in L2R only
> text), adding a newline where Emacs wraps a line, leads to "the same
> display" (modulo line-wrapping glyphs in the fringe and things like
> that, obviously).  And I understand Ehud's argument as saying that the
> same should hold for R2L and mixed text.
> 
> It seems at least a reasonable point of view.  Is your point of view
> based on the fact that it seems difficult/impossible to implement, or is
> it based on the fact that you think Ehud's expectation is weird?

I'm sorry I didn't make myself clear enough on that, to the degree
that such a question should at all be asked.

Like Ehud, I think that it would be swell to have what he wants.  But,
possibly unlike Ehud, I think that what I have now it not a disaster,
and we can live with it for the time being, maybe even longer.

The reasons for my decision to implement truncation and continuation
as I did are:

  . It is the only reasonable way to go that does not call for a very
    serious surgery, perhaps even a total rewrite, of the display
    engine code.

  . I saw no other editor that supports truncation and behaves
    otherwise.  (I don't know about any editors that support
    continuation lines like Emacs does.)  See below.

  . The issue pops up only in relatively rare situations: mixed
    L2R/R2L text that gets truncated/continued within a stretch of
    text whose directionality is against the paragraph direction.

> If it's just "difficult", then (just like rigid scrolling), it can be
> kept as a known shortcoming.

It is either VERY difficult or very slow.

The current display code lays out glyphs in each ``glyph row'' one by
one, in the visual order.  Thus, for the portion of text that is
reversed from its logical order, the bidi reordering code effectively
delivers the characters backwards to this glyph layout code, in the
decreasing order of buffer positions.  That is, the glyphs assembled
first are the last ones to be read.  Then you hit the window margin,
and know that there isn't enough place for the whole line.  Only then
you know how many characters will fit on this line.  But you know that
in terms of the last portion of the text in the reading order, which
tells you very little about how many characters at the beginning of
this stretch of text you could display instead.  (Remember that Emacs
supports variable size characters and different fonts on the same
line, so just counting characters will not do.)

What would be nice is to scan the text to be reversed in the logical
order, and find the part of it that will fit on this screen line.
Then we could reorder only that part.  But to do that, we need to try
every possibility by actually doing most of the display work behind
the scenes, because of the complications with different font sizes,
faces, composite characters and issues like ligatures and the like,
which change the amount of screen estate taken by a portion of a line,
even if you just juxtapose the same two characters.

With a newline marking the end of the line, it's easy: the bidi
reordering ends at the newline, then restarts after it.  By contrast,
to support ``bidi-smart continuation'', we need to find the place
where to break the line, and that is impossible without actually
trying to display it.

In the example below

  word1 word2 WORD1 WORD2

to be displayed as 

  word1 word2 2DROW 1DROW

if the window is only wide enough to display

  word1 word2  1DROW

we need to try displaying in order

  word1 word2 1
  word1 word2 1D
  word1 word2 1DR
  word1 word2 1DRO
  word1 word2 1DROW
  word1 word2  1DROW
  word1 word2 W 1DROW

until we discover where we should stop.  (We could do a binary search,
of course, but that's details.)  I don't think that's reasonable, and
I have no idea what will this do to the redisplay speed.

> How do other line-wrapping text editors with bidi support behave with
> such long mixed-L2R-R2L lines?

MS Word truncates in the visual order, like I suggest.  It doesn't
have continuation lines (it re-flows instead, which is the same as our
fill-paragraph, and that works like Ehud wants already.)  I didn't
have a chance of checking OpenOffice; I can do that next week, if no
one beats me to it.

I have Yudit installed, but I cannot get it to display Hebrew text (it
shows hex numbers instead); if someone knows how to do that on
Windows, please tell.  But someone told in this thread that it, too,
re-flows.  If that's true, we have no prior art that's different from
what I have now in the bidi Emacs.

Of course, Emacs breaks ground where many other similar tools punt...