bug#3687: 23.1.50; inconsistency in multibyte eight-bit regexps [PATCH]

[Eli Zaretskii]

> From: Mattias Engdegård <mattiase@acm.org>
> Date: Fri, 28 Jun 2019 16:05:07 +0200
> Cc: mituharu@math.s.chiba-u.ac.jp, monnier@iro.umontreal.ca,
>         3687@debbugs.gnu.org
> 
> > 1. What do you mean by "raw bytes"?  Is #xab a raw byte or a Unicode
> >    point U+00AB?  IOW, how do we distinguish, in a regexp, between a
> >    raw byte and a character whose Unicode codepoint is that byte's
> >    value?  And how does one go about concocting a regexp that matches
> >    raw bytes in a unibyte or multibyte buffer or string?
> 
> Sorry, I should have been more clear. The terminology in the manual is a bit 
> muddled; in this case I mean the characters (or whatever you prefer calling 
> them) obtained with hex or octal escapes in the range 128-255, such as "\xff" 
> or "\377", regardless of the string's type (unibyte or multibyte).
> 
> Unicode characters in the range 128-255 can be generated using the \u00HH or 
> \U000000HH notations, or by just including them literally. They are distinct 
> from raw bytes.
> 
> To match raw bytes, just write them. They are not special in regexp syntax 
> and need no escaping.

So this means \240 is no longer the same as NBSP and \300 is no longer
the same as À?  But \176 is still the same as ~?  Doesn't this open a
clear path for another bug report about inconsistencies in regexps?

Also, which ways do you propose for specifying raw bytes?  Only hex
escapes? octal escapes as well? something else?

> > 2. What is meant by "ranges from ASCII to raw bytes"?  Which
> >    characters are included in such ranges?
> 
> Ranges such as [A-\xb1] or [\000-\377], where the first endpoint is an ASCII 
> character and the last endpoint is a raw byte as defined above. These should 
> include all characters from the first endpoint up to and including ASCII 127, 
> and all raw bytes from 128 to the last endpoint. This makes intuitive sense 
> for unibyte strings where such an interval is contiguous in the underlying 
> representation; extending them to multibyte is obvious.

So you are saying that we will consider the raw bytes as if they
followed ASCII characters in the lexicographical order?  But non-ASCII
characters whose codepoints start at 0x80? where are they in this
order?

> > 3. If ranges from non-ASCII characters to raw bytes make no sense,
> >    how would one go about specifying a range that includes all the
> >    characters and raw bytes supported by Emacs?
> 
> "[\x00-\U0010ffff\x80-\xff]"

This looks confusing, because to a naïve reader the first part already
includes the second one.

My point is that I'm afraid this proposal will replace one set of
inconsistencies by another.

I think the only way to avoid inconsistencies is to consider the likes
of \177 mean different things depending on whether the text being
matched is unibyte or multibyte.  In particular, raw bytes in
multibyte regexps should (if they are needed) be spelled out as
#x3fff00, \17777400, etc.  This, of course, has the disadvantage that
one needs to know which text is being matched before one concocts the
regexp.