Re: Syntax-ppss flush should happen in after-change hook

[Stefan Monnier]

> Currently ppss flush happens in before-change hook. This leads to a real
> possibility that syntax-propertize is called before the actual change and not
> refreshed after the change. This could occur in at least two scenarios.
>
> First, when a hook placed after syntax-ppss-flush-cache in
> before-change-functions invokes `syntax-propertize` (for example with a
> search).

If syntax-ppss-flush-cache is on after-change-functions then the risk is
when a hook function is placed before it.  Given the fact that add-hook
defaults to placing functions at the head and that
after-change-functions are much more commonly used than
before-change-functions, the risks associated with placing it on
after-change-functions are at least as high.

FWIW, syntax-ppss-flush-cache was originally placed on
after-change-functions and later changed to before-change-functions
because of exactly that problem.

> Second is when the actual function responsible for a change invokes
> `syntax-propertize`.
>
> Here is a real case of uncomment-region in markdown-mode:
>
>   markdown-syntax-propertize(63 823)
>   syntax-propertize(823)
>   internal--syntax-propertize(64)
>   parse-partial-sexp(63 #<marker at 77 in test.tmd> nil nil nil t)
>   comment-search-forward(#<marker at 77 in test.tmd> t)
>   uncomment-region-default-1(63 77 nil)
>   #f(compiled-function () #<bytecode 0x280781d>)()
>   combine-change-calls-1(63 77 #f(compiled-function () #<bytecode 0x280781d>))
>   uncomment-region-default(63 77 nil)
>   uncomment-region(63 77 nil)

I do see that combine-change-calls is involved and indeed
combine-change-calls seems just fundamentally problematic in this
respect (the problem would appear as well if we moved
syntax-ppss-flush-cache on after-change-functions since it would mean
that the syntax-ppss cache is left inconsistent during the execution of
its body).

The interaction between the syntax-ppss cache (and the syntax-propertize
properties as well) and the changes that can take place during
combine-change-calls can't be done right with a coarse "run something at
the beginning and something else at the end".

A crude solution could be the ad-hoc hack below.  We could make it less
ad-hoc by adding a way for any hook function (on b-c-f and a-c-f) to
indicate whether it should stay active within combine-change-calls,
e.g. with a symbol property `do-not-combine-change-function`.


        Stefan


diff --git a/lisp/subr.el b/lisp/subr.el
index 3a901b3174..abdc116e5f 100644
--- a/lisp/subr.el
+++ b/lisp/subr.el
@@ -3834,7 +3834,10 @@ combine-change-calls-1
        (if (eq buffer-undo-list t)
            (setq result (funcall body))
          (let (;; (inhibit-modification-hooks t)
-                before-change-functions after-change-functions)
+                (before-change-functions
+                 (if (memq #'syntax-ppss-flush-cache before-change-functions)
+                     '(syntax-ppss-flush-cache)))
+                after-change-functions)
            (setq result (funcall body)))
          (let ((ap-elt
                 (list 'apply