[O] bug#11700: bug#11700: 24.1.50; Bad interaction between BiDi and org-tables

[Eli Zaretskii]

> Date: Fri, 15 Jun 2012 09:39:35 +0300
> From: Dov Grobgeld <address@hidden>
> Cc: address@hidden
> 
> Yes. Great! This is indeed what I wanted. My mistake was that I tried
> it with a tab character before OR after the vertical bar. This
> solution should be really simple to implement in org-mode as it means
> that instead of joining the table columns with "<space><vbar><space>"
> as is currently done, you just need to use "<tab><vbar><tab>" as well
> as setting the tab width to 1.

Yep.

> But I just wonder, is there any other character (preferably white
> space character) with the same end-of-segment-boundary properties as
> tab, in case tab is used for something else in org-mode?

That's the (space . :align-to COLUMN) display property I was talking
about.  With it, you can arrange for a single blank, say, to produce a
stretch of whitespace of arbitrary size, in character cell units,
aligned to a specified column.  If you use :width instead of
:align-to, you can have the size tuned in pixels.  Emacs treats text
covered by such display properties as segment separators, so they
produce the same effect as TAB does.  That's because conceptually,
such display properties are used to separate text into columnar
display, exactly like TAB is used in plain-text tables.

You can find examples of using these display properties in
minibuffer.el, where they are used to produce the display in the
*Completions* buffer.  Their documentation is in the ELisp manual.

> Or is it possible to take an arbitrary character, e.g. U+E0020, and
> bless it with end-of-segment boundary properties and then use that
> in the org-mode buffer?

Try using u+2029, it should do the trick, I think.

> As a side note, I really like the idea of end of segment separator,
> and I wasn't aware of it when I wrote fribidi a long time ago. I
> wonder if the semantics of the emacs segment separator follows the
> Unicode Bidi Algorithm?

Of course, it does; Emacs implements the UBA to the letter, taking
only the "high-level protocols" fire-escape to guide the reordering using
Emacs-specific context.  (But even the high-level protocols feature is
part of the UBA, see section 4.3 there.)

The Segment Separator feature is not an Emacs invention, it exists in
the UBA.  The key to understanding it is this part of the UBA:

  L1. On each line, reset the embedding level of the following
      characters to the paragraph embedding level:

  1. Segment separators,
  2. Paragraph separators,
  3. Any sequence of whitespace characters preceding a segment
     separator or paragraph separator, and 
  4. Any sequence of white space characters at the end of the line.

And the TAB character has "S", i.e. "segment separator", as its bidi
property.  Since its embedding level is reset to the base embedding
level of the paragraph, the result is that text on both sides of a TAB
is reordered separately and independently.

Moreover, the high-level protocols feature give us one more
possibility:

  HL4. Apply the Bidirectional Algorithm to segments.

    . The Bidirectional Algorithm can be applied independently to one
      or more segments of structured text. For example, when
      displaying a document consisting of textual data and visible
      markup in an editor, a higher-level process can handle syntactic
      elements in the markup separately from the textual data.

Emacs uses this to treat the `space' display properties as segment
separators.