bug#30393: 24.4; cperl-mode: indentation failure - Documentation enhancements

[Alan Mackenzie]

Hello, Eli.

On Mon, Apr 09, 2018 at 21:41:21 +0300, Eli Zaretskii wrote:
> > Date: Sun, 8 Apr 2018 10:52:57 +0000
> > Cc: monnier@IRO.UMontreal.CA, 30393@debbugs.gnu.org, dgutov@yandex.ru,
> >   npostavs@users.sourceforge.net
> > From: Alan Mackenzie <acm@muc.de>

> > Can we start moving forward with this change, please?

> What prevents you from moving forward with it?  You already know what
> needs to be done to move forward: documentation of the new primitives
> and some tests.

After mulling it over for several weeks, I now think it would be better
just to leave these new primitives out.  Although they might be useful,
their main use was for me whilst hacking the syntax routines.

However, I propose the following documentation changes to go with my
patch to the code, these changes clarifying some of the limitations
inherent to syntax-ppss, and indicating how my enhancements will work.



diff --git a/doc/lispref/syntax.texi b/doc/lispref/syntax.texi
index 3327d7855c..b58e2a9a98 100644
--- a/doc/lispref/syntax.texi
+++ b/doc/lispref/syntax.texi
@@ -430,6 +430,11 @@ Syntax Table Functions
 This function always returns @code{nil}.  The old syntax information in
 the table for this character is discarded.
 
+Note that if the modification of @var{table} changes the way it parses
+comments or strings, the cache used by @code{syntax-ppss} will be
+emptied in each buffer whose current syntax table (or its parent,
+etc.) is @var{table}.  @xref{Position Parse}.
+
 @example
 @group
 @exdent @r{Examples:}
@@ -502,6 +507,12 @@ Syntax Table Functions
 @defun set-syntax-table table
 This function makes @var{table} the syntax table for the current buffer.
 It returns @var{table}.
+
+Note that if @var{table} parses comments or strings differently from
+the buffer's previous syntax table, the cache used by
+@code{syntax-ppss} in this buffer will be emptied.  @xref{Position
+Parse}.
+
 @end defun
 
 @defun syntax-table
@@ -523,6 +534,9 @@ Syntax Table Functions
 more precise: @code{with-syntax-table} temporarily alters the current
 syntax table of whichever buffer is current at the time the macro
 execution starts.  Other buffers are not affected.
+
+This macro might empty the cache used by @code{syntax-ppss}, as noted
+above under @code{set-syntax-table}.
 @end defmac
 
 @node Syntax Properties
@@ -534,6 +548,15 @@ Syntax Properties
 occurrences in the buffer, by applying a @code{syntax-table} text
 property.  @xref{Text Properties}, for how to apply text properties.
 
+As an alternative to setting @code{syntax-table} text properties
+directly, you can use @code{syntax-propertize-function} (see below).
+Most major modes supplied with Emacs which use these text properties
+use this method for applying them.  We strongly recommended you to use
+just one of these methods in any Emacs Lisp program, and not to mix
+them in the same buffer.@footnote{@code{syntax-propertize-function}
+can operate at unpredictable times, and may erase explicitly applied
+@code{syntax-table} properties.}
+
   The valid values of @code{syntax-table} text property are:
 
 @table @asis
@@ -556,6 +579,10 @@ Syntax Properties
 If this is non-@code{nil}, the syntax scanning functions, like
 @code{forward-sexp}, pay attention to syntax text properties.
 Otherwise they use only the current syntax table.
+
+If you change this variable after buffer initialization time, even if
+your Emacs Lisp program doesn't use @code{syntax-ppss}, you should
+call @code{syntax-ppss-flush-cache}.  @xref{Position Parse}.
 @end defvar
 
 @defvar syntax-propertize-function
@@ -695,8 +722,9 @@ Motion via Parsing
 negative @var{depth} has the effect of moving deeper by @var{-depth}
 levels of parenthesis.
 
-Scanning ignores comments if @code{parse-sexp-ignore-comments} is
-non-@code{nil}.
+Scanning skips over comments if @code{parse-sexp-ignore-comments} is
+non-@code{nil} (which it usually is for programming language major
+modes).
 
 If the scan reaches the beginning or end of the accessible part of the
 buffer before it has scanned over @var{count} parenthetical groupings,
@@ -709,7 +737,7 @@ Motion via Parsing
 It returns the position where the scan stops.  If @var{count} is
 negative, the scan moves backwards.
 
-Scanning ignores comments if @code{parse-sexp-ignore-comments} is
+Scanning skips over comments if @code{parse-sexp-ignore-comments} is
 non-@code{nil}.
 
 If the scan reaches the beginning or end of (the accessible part of) the
@@ -747,7 +775,7 @@ Position Parse
 
   For syntactic analysis, such as in indentation, often the useful
 thing is to compute the syntactic state corresponding to a given buffer
-position.  This function does that conveniently.
+position.  @code{syntax-ppss} does that conveniently.
 
 @defun syntax-ppss &optional pos
 This function returns the parser state that the parser would reach at
@@ -769,6 +797,33 @@ Position Parse
 complete subexpression) and sixth value (minimum parenthesis depth) in
 the returned parser state are not meaningful.
 
+Note that these caches become invalid when you set a new syntax table
+for the buffer, or change an entry for a character in the current
+syntax table.  If you set or clear a @code{syntax-table} text property
+at some buffer position, the caches become invalid for the buffer
+portion at and after that position.  It is your responsibility to deal
+with these situations, either by calling
+@code{syntax-ppss-flush-cache} (see below), or by refraining from
+using @code{syntax-ppss} while the caches are invalid.  An exception
+to this is when such a change alters the way comments or strings are
+parsed; then, Emacs calls @code{syntax-ppss-flush-cache}
+automatically.@footnote{The reason for this is that from Emacs 27,
+Emacs uses @code{syntax-ppss} internally in low level primitives such
+as @code{forward-comment} and @code{scan-lists}.  This automatic
+flushing of the cache helps these primitives continue to work
+reliably.}
+
+Changing any of the variables @code{multibyte-syntax-as-symbol},
+@code{parse-sexp-ignore-comments}, @code{comment-end-can-be-escaped}
+(@pxref{Control Parsing}), or @code{parse-sexp-lookup-properties}
+(@pxref{Syntax Properties}) after buffer initialization will always
+leave @code{syntax-ppss}'s caches invalid in the affected buffers.  So
+will changing a symbol's @code{syntax-table} property when that symbol
+is the value of a @code{category} text property somewhere in the
+buffer (@pxref{Special Properties}), a practice we don't recommend.
+In such cases you must always take one of the actions detailed in the
+previous paragraph.
+
 This function has a side effect: it adds a buffer-local entry to
 @code{before-change-functions} (@pxref{Change Hooks}) for
 @code{syntax-ppss-flush-cache} (see below).  This entry keeps the
@@ -952,6 +1007,11 @@ Control Parsing
 You can use @code{forward-comment} to move forward or backward over
 one comment or several comments.
 
+If you change any of the above three variables after buffer
+initialization time, even if your Emacs Lisp program doesn't use
+@code{syntax-ppss}, you should call @code{syntax-ppss-flush-cache}.
+@xref{Position Parse}.
+
 @node Syntax Table Internals
 @section Syntax Table Internals
 @cindex syntax table internals
diff --git a/doc/lispref/text.texi b/doc/lispref/text.texi
index ebfa8b9b0f..d8df9fe352 100644
--- a/doc/lispref/text.texi
+++ b/doc/lispref/text.texi
@@ -3203,6 +3203,14 @@ Special Properties
 properties of this symbol serve as defaults for the properties of the
 character.
 
+You might be tempted to put a @code{syntax-table} property onto the
+symbol, and change this property's value repeatedly in your Lisp
+program, thus changing the syntax of many characters in a buffer at
+the same time.  We advise against doing this, since it renders the
+caches used by @code{syntax-ppss} invalid in a way that Emacs cannot
+detect and correct for.  If you are going to be doing this, please
+consult @ref{Position Parse} and follow the advice there.
+
 @item face
 @cindex face codes of text
 @kindex face @r{(text property)}


-- 
Alan Mackenzie (Nuremberg, Germany).