[O] Fwd: org-mode and 'intangible text property

[ISHIKAWA,chiaki]

Hi, I have posted this to help-gnu-emacs mailing list, but was advised to org-mode specific mailing list and so am 
posting this here.

After posting, I thought maybe the use of overlay property instead of 
text-property might be the way to go.
But I would appreciate the opinion of the people in the know.

Thank you.


-------- Original Message --------
Subject: org-mode and 'intangible text property
Date: Thu, 18 Apr 2013 19:07:48 +0900
From: address@hidden
To: <address@hidden>

org-mode and 'intangible text property

Hello,

I have a question about org-mode.el.
(sorry for the lengthy posting, I wanted to list the
background of the issue in one e-mail.)

I am a user of Emacs over the years (20+ years).
I have been using outline mode and have come to realize that org-mode
seems to offer additional benefits by reading blogs and others.

However, when I tried to use org-mode, I hit a snug.

I tried to input Japanese string using a Japanese input system written
in Emacs Lisp called tamago/egg.  But when I tried to type in, say, a
first-level heading in org-mode, the string completely disappears (not
even a trace of it) after I hit the return (to finalize the Japanese
input.)

The cause of the problem seems to be as follows:
The use of 'intangible property of org-mode collided with the
use of 'intangible property of the Japanese input method
implemented in Emacs Lisp.

Basically, the problem is that org-mode removes the 'intangible
property set by egg/tamago Japanese input system without
discrimination at all is what someone wrote in his blog. (org-mode
seems to use 'intangible internally.)

The Japanese input system, egg/tamago, was a rather popular system in
late 1980 and 1990's. It now sees many competitive Japanese input
systems, but there are emacs die hards who can't live without it when
it comes to typing Japanese into Emacs buffer.

Although egg/tamago is in strictly maintenance mode, it is difficult
for me (and others who have grown up to use this input system
figuratively speaking) to ditch this egg/tamago input system if I can
avoid it.
[No major update has been done in the last 4-5 years to tamago/egg and
its major web site is now torn down. Only the packages that are in the
major linux distributions, FreeBSD, NetBSD packaging systems are more
or less used by many, with some patches floating around. For example,
when emacs 23.4 appeared, finally we have to replace or use defalias
to fix the missing last-command-char variable in egg/tamago package.
There *ARE* people willing to maintain egg/tamago to keep it alive in
the face of such
issues for sure, mine included.]

On the other hand, if org-mode is a great replacement for
outline-mode, then I would certainly like to try it out
(and still use egg/tamago for input if possible.)


Org mode seems to use 'intangible text property one way or the other.
igrep mode search within org subdirectory of lisp directory of emacs
24.3 turned up the following hits (shown at the end of this e-mail).

One of the macro's found in  org-mac.el is as follows. :
org-maybe-intangible
This is quoted a few lines below.

The document string for this macro suggests that org-mode had an issue
with flyspell.el due to the intangible text property use. But now the
particular usage seems to be avoided by the introduction of macros.

;; FIXME: Slated for removal.  Current Org mode does not support Emacs
< 22
(defmacro org-maybe-intangible (props)
  "Add '(intangible t) to PROPS if Emacs version is earlier than Emacs
22.
In Emacs 21, invisible text is not avoided by the command loop, so the
intangible property is needed to make sure point skips this text.
In Emacs 22, this is not necessary.  The intangible text property has
led to problems with flyspell.  These problems are fixed in
flyspell.el,
but we still avoid setting the property in Emacs 22 and later.
We use a macro so that the test can happen at compilation time."
  (if (< emacs-major-version 22)
      `(append '(intangible t) ,props)
    props))


HOWEVER there are still about a dozen references to
'intangible property in org-mode emacs-lisp code.

I wonder if such avoidance as in the case of flyspell.el can be made
for the conflict of usage of 'intangible with egg/tamago input routine
written in Emacs-Lisp.

Someone in Japan who faced this incompatibility problem of org-mode
and egg/tamago input method stated that he (I assume he) substituted
all the references to "intangible" in org-mode elisp code with
"org-intangible". Now, as far as he is concerned, the Japanese input
now
works, and he has not bumped into a case where cursor moves into an
awkward (or should not happen) position in org-mode (which should have
been avoided by the use of 'intangible property in org-mode).

His drastic surgery may be the last straw if no alternative is
available.  (His blog in Japanese is available here:
http://d.hatena.ne.jp/grandVin/20090129/1233226833
(dated January 2009)

Also, it is quoted in an independent someone's emacs wiki written in
Japanese
prominently.  http://www.tsukune.org/skk/memo/index.php?emacs#n27db692

At least you can see there are Japanese users who are bitten by the
problem and want to solve it somehow.

Does anyone in the know can evaluate the perspective of removing the
references to 'intangible in the org code?  If, due to the conflict
with flyspell.el, some usages of intangible could be removed, can some
remaining references be removed, too?  (Well, I am not sure of
this. Probably intangible is used to make sure that the cursor does
not move, say, into a next cell in a table.)

OR, more important to me, can someone suggest a manner how a
conflicting use of
'intangible can be somehow managed?

Thank you for reading so far.

I am outlining a rough idea how this could be handled. But maybe I am
wrong.

Any tips will be appreciated.

TIA


PS:  my idea is to use defadvise
function to redefine the meanings of the following functions both in
org-mode and when egg/tamago Japanese input is in effect,

- set-text-properties
- remove-text-properties
- add-text-properties (may not be necessary according to the usage
pattern.)
- get-text-property
- put-text-property

These functions are used to set/add or reference the text property in
org-mode (and egg/tamago Japanese input method.)

My idea is roughly as follows:

Let us keep the both forms of 'intangible property PLUS the value pair
for
org-mode and egg/tamago as part of the text property, and
set/remove and return only the property set in the same mode.  Let us
use
org-mode-intangible and egg-mode-intangible for these internal forms in
the explanation below.

For example, when remove-text-properties is called and 'intangible is
about to be removed, and if it is called within org-mode, delete the
intangible property for org-mode, i.e., org-mode-intangible becomes
nil.  if it is called from egg/tamago input mode, intangible property
for egg-mode, i.e., egg-mode-intangible becomes nil.
(If it is called outside these modules, the behavior of the function
remains the same.)

When get-text-property is called for 'intangible, check what was set
in each mode previously, and return the value for the matching mode.
I found that, for org-mode, we can simply return t or nil for
'intangible.
For egg-mode, we probably need to return 'its-part-1, 'its-part-2 or
nil depending on
what was set before.
(If it is called outside these modules, the behavior of the function
remains the same.)

Now the difficult part is to set 'intangible that is shown to the rest
of the Emacs Lisp world (outside org-mode and egg/tamago input mode).
'intangible property seems to be used by display code written in C and
so we must produce something that is visible to such routines.
We need to set 'intangible to non-nil or nil depending on what was set
in org-mode and egg-mode (probably the LOGICAL OR of what was set
previously in both modes).

I have no idea if my idea above works or not.

The drastic surgery of replacing 'intangible in org-mode lisp code to
'org-intangible mentioned in someone's blog is a desire to keep the
operation of
the particular Japanese input system intact over the possible rare
misbehavior of
org-mode.  But if we can fix both elisp codes somehow to make
them compatible as in the case of flyspell.el, that will be wonderful.

Any tips will be appreciated.


Here is the search for "intangible" in  org-mode Emacs Lisp code.

grep -n  -e intangible *.el /dev/null
org-colview.el:248:     (overlay-put ov 'intangible t)
       CI's comment:  this usage may be a little tricky to handle.

org-macs.el:151:(defmacro org-maybe-intangible (props)
org-macs.el:152:  "Add '(intangible t) to PROPS if Emacs version is
earlier than Emacs 22.
org-macs.el:154:intangible property is needed to make sure point skips
this text.
org-macs.el:155:In Emacs 22, this is not necessary.  The intangible
text property has
org-macs.el:160:      `(append '(intangible t) ,props)

       CI's comment: the above is the org-maybe-intangible defintion.
org-macs.el:239:(defconst org-rm-props '(invisible t face t keymap t
intangible t mouse-face t
          CI's comment:
          org-rm-props is used in the following
          places, but their usage are as args to
          the few set of functions:

          grep -n  -e org-rm-props ~/repos/emacs-24.3/lisp/org/*.el /dev/null
          ~/repos/emacs-24.3/lisp/org/org-macs.el:239:(defconst
org-rm-props '(invisible t face t keymap t intangible t mouse-face t
          ~/repos/emacs-24.3/lisp/org/org-macs.el:247:  (and s
(remove-text-properties 0 (length s) org-rm-props s))
          CI's comment: the above is used as an arg to remove-text-properties)
          ~/repos/emacs-24.3/lisp/org/org-macs.el:254:in `org-rm-props'."

          The above is the usage in the following: the following
          processing may suggest one way to deal with the subtle issue
          of usage conflicts.

        (defsubst org-no-properties (s &optional restricted)
          "Remove all text properties from string S.
        When RESTRICTED is non-nil, only remove the properties listed
        in `org-rm-props'."
          (if (fboundp 'set-text-properties)
              (set-text-properties 0 (length s) nil s)
            (if restricted
                (remove-text-properties 0 (length s) org-rm-props s)
              (set-text-properties 0 (length s) nil s)))
          s)

          ~/repos/emacs-24.3/lisp/org/org-macs.el:258:  (remove-text-properties
0 (length s) org-rm-props s)
          ~/repos/emacs-24.3/lisp/org/org.el:8018:  (remove-text-properties 0
(length s) org-rm-props s)
          CI's comment: the above is used as an arg to remove-text-properties)
          The above usages are as args to remove-text-properties.


org-src.el:314:                         '(display nil invisible nil intangible 
nil))
    (CI's comment: this is used as an arg to remove-text-properties)

org-src.el:472:                         '(display nil invisible nil intangible 
nil))
    (CI's comment: the above is used as an arg to
remove-text-properties)

org-table.el:1894:                                             display t 
intangible t))
    (CI's comment: the above is used as an arg to
remove-text-properties)

org-table.el:1923:                                        intangible t))
    (CI's comment: the above is used as an arg to
remove-text-properties)

org.el:5301:  '(mouse-face highlight keymap invisible intangible
help-echo org-linked-text htmlize-link))

    CI's comment: the above is used in  (defconst org-nonsticky-props
  '(mouse-face highlight keymap invisible intangible help-echo
org-linked-text htmlize-link))

org.el:5328:                            '(display t invisible t intangible t))
    (CI's comment: the above is used as an arg to
remove-text-properties)

org.el:5370:                                '(display t invisible t intangible 
t))
    (CI's comment: the above is used as an arg to
remove-text-properties)

org.el:5389:                                    '(display t invisible t 
intangible t)))
    (CI's comment: the above is used as an arg to
remove-text-properties)


org.el:5490:         (ip (org-maybe-intangible
     CI's comment org-maybe-intangible does not use 'intangible in
     emacs 24, I think.

org.el:5572:      (add-text-properties s e (org-maybe-intangible
'(invisible org-cwidth)))
     CI's comment org-maybe-intangible does not use 'intangible in
     emacs 24, I think.

org.el:5987:                                     invisible t intangible t
     CI's comment: the above is used as an arg to
     remove-text-properties,
     i.e.   (remove-text-properties beg end
                            '(mouse-face t keymap t org-linked-text t
                                         invisible t intangible t
                                         org-no-flyspell t org-emphasis t))


org.el:10194:   (unless (get-text-property (match-end group) 'intangible)
; Emacs 21
     CI's comment: checking with the get-text-property at the position.

Igrep finished (matches found) at Thu Apr 18 16:45:43


[] Searching the usage of "intangible" in tamago/egg

./egg-cnv.el:185:(defmacro egg-bunsetsu-info () ''intangible)

./its.el:326:                                'intangible         'its-part-2
    CI's comment: The above is part of set-text-properties as in

      (set-text-properties 0 1 (list 'read-only          t
                                     'invisible          'egg
                                     'intangible         'its-part-2
                                     'its-cursor         cursor
                                     'point-entered      'egg-enter/leave-fence
                                     'point-left         'egg-enter/leave-fence
                                     'modification-hooks '(egg-modify-fence))
                           str)
      (insert str)

./its.el:350:;; intangible intangible
./its.el:354:  (let ((open-props '(its-start t intangible its-part-1))
          CI's comment: open-props is used subsequently
          as an argument to add-text-properties.
          (add-text-properties p p1 open-props)

 ./its.el:355:  (close-props '(rear-nonsticky t its-end t intangible
its-part-2))
          CI's comment: close-props is used subsequently
          as an argument to add-text-properties.
      (add-text-properties p1 (point) close-props)

./its.el:388:    (put-text-property 0 (length str) 'intangible
'its-part-1 str)
       CI's comment : the above is as argument to put-text-property.
./its.el:579:                          'intangible 'its-part-1)
       CI's comment: the above is as argument to add-text-property

    (add-text-properties 0 (length output)
                         (list 'its-map its-current-map
                               'its-syl newsyl
                               'egg-lang its-current-language
                               'read-only t
                               'intangible 'its-part-1)
                         output)

./its.el:895:    (put-text-property 0 (- end beg) 'intangible
'its-part-1 str)
./its.el:903:    (put-text-property 0 (- end beg) 'intangible
'its-part-2 str)
./its.el:1167:    (remove-text-properties 0 len '(intangible nil)
source)
./its.el:1197:    (add-text-properties 0 len '(read-only t intangible
its-part-1) source)
./its.el:1199:      (add-text-properties 0 len '(read-only t intangible
its-part-2) source)
./its.el:1274:  (and (eq (get-text-property (point) 'intangible)
'its-part-2)
./its.el:1527:      (put-text-property 0 len 'intangible 'its-part-1
old-str)


TIA