bug#68446: 29.1.90; Bidi right-to-left paragraphs missing text in Org mode

[Eli Zaretskii]

> From: Ihor Radchenko <yantar92@posteo.net>
> Cc: Thamer Mahmoud <thamer.mahmoud@gmail.com>, 68446@debbugs.gnu.org
> Date: Sun, 14 Jan 2024 12:11:03 +0000
> 
> Eli Zaretskii <eliz@gnu.org> writes:
> 
> > I think it's an Org bug: it should prevent bidi reordering inside the
> > "[[link]]" string.  For example, wrap the "[[link]]" thing in LRO..PDF
> > bidi controls.  Because without that, the brackets can be mirrored by
> > bidi reordering and the BPA algorithm, and the link is no longer in
> > the form that Org expects.  The result is that the entire text becomes
> > invisible.
> 
> May you please elaborate? I do not see anything unexpected in the text
> properties that Org mode applies to the link and the letter "a".

(For some value of "expected".  Me, I don't understand more than half
of the properties there, wrt their expected effect on the text and its
display.  Are you surprised?)

Anyway, after performing the recipe, do this:

  . M-<
  . C-x =
  . C-f
  . C-x =
  . C-f
  . C-x =

etc.  IOW, move 1 buffer position forward at a time from BOB, and each
time show the character at that position.  Do you see something
strange?

Then do the same in a buffer under Fundamental mode with the same text
and the same value of bidi-paragraph-direction.  Can you explain what
Org does to cause the difference, which makes some characters be
"skipped" by C-f (due to point-adjustment feature), and probably also
makes all the characters invisible as result?

I can tell you what I see when stepping through the display code: it
skips all the characters as invisible.  I see that org-link is in the
buffer-invisibility-spec, but that's probably just the tip of the
iceberg, so I'm asking you below to describe what are all those faces
and properties for, and how they are related to the invisibility.

> In fact, after following the reproducer, and enabling visible-mode, M-x
> describe-text-properties on "a" yields
> 
>     Text content at position 101:
>     
>     
>     There are text properties here:
>       fontified            t
> 
> Yet, "a" is rendered invisible.

When you invoke visible-mode, the text is magically shown as expected.
Can you explain why is that?

If you invoke describe-text-properties on any character in the buffer,
there's no character with 'invisible' face, either.  And yet they are
invisible.  How come?

If you describe in enough details how the [[link]] thing is handled by
Org, maybe I could elaborate more than the general observations and
questions above.

To summarize, my point is simple: the Emacs bidirectional display is
responsible for reordering plain text with faces, and while doing so,
it more-or-less ignores invisible text.  Anything fancier: overlays,
images, text properties that affect display in some other ways,
etc. -- all these are not guaranteed to behave in reordered text as
they do in strict L2R text, and Lisp programs which create visual
effects using those fancier features should look out for problems, and
should use bidi formatting controls judiciously to prevent such
disasters.

(My guess is that the brackets in this case are reordered, and as
result cause everything to become invisible.  But that's just a guess,
and I don't expect to understand why without a lot more background and
help from Org experts.)