Re: [patch suggestion] Mitigating the poor Emacs performance on huge org files: Do not use overlays for PROPERTY and LOGBOOK drawers

[Nicolas Goaziou]

Ihor Radchenko <address@hidden> writes:

> This should be better:
> https://gist.github.com/yantar92/e37c2830d3bb6db8678b14424286c930

Thank you.

> This might get tricky in the following case:
>
> :PROPERTIES:
> :CREATED: [2020-04-13 Mon 22:31]
> <region-beginning>
> :SHOWFROMDATE: 2020-05-11
> :ID:       e05e3b33-71ba-4bbc-abba-8a92c565ad34
> :END:
>
> <many subtrees in between>
>
> :PROPERTIES:
> :CREATED:  [2020-04-27 Mon 13:50]
> <region-end>
> :ID:       b2eef49f-1c5c-4ff1-8e10-80423c8d8532
> :ATTACH_DIR_INHERIT: t
> :END:
>
> If the text in the region is replaced by something else, <many subtrees
> in between> should not be fully hidden. We cannot simply look at the
> 'invisible property before and after the changed region. 

Be careful: "replaced by something else" is ambiguous. "Replacing" is an
unlikely change: you would need to do 

  (setf (buffer-substring x y) "foo")

We can safely assume this will not happen. If it does, we can accept the
subsequent glitch.

Anyway it is less confusing to think in terms of deletion and insertion.
In the case above, you probably mean "the region is deleted then
something else is inserted", or the other way. So there are two actions
going on, i.e., `after-change-functions' are called twice.

In particular the situation you foresee /cannot happen/ with an
insertion. Text is inserted at a single point. Let's assume this is in
the first drawer. Once inserted, both text before and after the new text
were part of the same drawer. Insertion introduces other problems,
though. More on this below.

It is true the deletion can produce the situation above. But in this
case, there is nothing to do, you just merged two drawers into a single
one, which stays invisible. Problem solved.

IOW, big changes like the one you describe are not an issue. I think the
"check if previous and next parts match" trick gives us roughly the same
functionality, and the same glitches, as overlays.

However, I think we can do better than that, and also fix the glitches
from overlays. Here are two of them. Write the following drawer:

  :FOO:
  bar
  :END:

fold it and delete the ":f". The overlay is still there, and you cannot
remove it with TAB any more. Or, with the same initial drawer, from
beginning of buffer, evaluate:

  (progn (re-search-forward ":END:") (replace-match ""))

This is no longer a drawer: you just removed its closing line. Yet, the
overlay is still there, and TAB is ineffective.

Here's an idea to develop that would make folding more robust, and still
fast.

Each syntactical element has a "sensitive part", a particular area that
can change the nature of the element when it is altered. Luckily drawers
(and blocks) are sturdy. For a drawer, there are three things to check:

1. the opening line must match org-drawer-regexp
2. the closing line must match org-property-end-re (case ignored)
3. between those, you must not insert text match org-property-end-re or
   org-outline-regexp-bol

Obviously, point 3 needs not be checked during deletion.

Instead of `after-change-functions', we may use `modification-hooks' for
deletions, and `insert-behind-hooks' for insertions. For example, we
might add modification-hooks property to both opening and closing line,
and `insert-behind-hooks' on all the drawer. If any of the 3 points
above is verified, we remove all properties.

Note that if we can implement something robust with text properties, we
might use them for headlines too, for another significant speed-up.

WDYT?

> I think that using fontification (something similar to
> org-fontify-drawers) instead of after-change-functions should be
> faster.

I don't think it would be faster. With `after-change-functions',
`modification-hooks' or `insert-behind-hook', we know exactly where the
change happened. Fontification is fuzzier. It is not instantaneous
either.

It is an option only if we cannot do something fast and accurate with
`after-change-functions', IMO.