bug#66760: 29.1; [BUG] GB18030 Incorrect Encoding

[Eli Zaretskii]

> Date: Thu, 26 Oct 2023 19:43:54 +0800
> From: "Ruijie Yu" <yuruijie@sics.ac.cn>
> 
> Hello,
> 
> I have noticed that in GB18030 encoding, certain ranges of characters
> have incorrect encodings.
> 
> One example is U+217A (SMALL ROMAN NUMERAL ELEVEN).  The expected
> encoding is 81 36 C5 30 (as can be seen from the GB18030 standard [1]
> and verified from other programs such as iconv and MySQL), whereas the
> observed encoding within Emacs is 81 36 C4 39, with a 1-codepoint
> offset.
> 
> This behavior can be reproduced by the following recipe under both
> GNU/Linux and Windows:
> 
> --8<---------------cut here---------------start------------->8---
> $ emacs
> C-x h DEL
> C-x C-m f gb18030 RET
> C-x 8 RET 217a RET
> M-<
> C-u C-x =
> ;; observe the "file code":
> ;; file code: #x81 #x36 #xC4 #x39 (encoded by coding system 
> chinese-gb18030-dos)
> --8<---------------cut here---------------end--------------->8---
> 
> In contrast, this is what I get on MySQL (which I have also verified
> against the GB18030 standard):
> 
> --8<---------------cut here---------------start------------->8---
> > CREATE TABLE gb (id INT, c TEXT CHARACTER SET GB18030);
> > INSERT INTO gb VALUES (0, 'ⅺ');
> > SELECT HEX(c) FROM gb;
> 
> +----------+
> | hex(c)   |
> +----------+
> | 8136C530 |
> +----------+
> --8<---------------cut here---------------end--------------->8---
> 
> Beyond this, I also noticed that U+A642 (CYRILLIC CAPITAL LETTER DZELO)
> has the encoding 82 36 B9 36 on Emacs, whereas MySQL has 82 36 BA 35,
> which has an offset of 9 codepoints.
> 
> Could someone with more expertise and time look into why there is a
> mismatch between Emacs' GB18030 data and the standard?

Alas, we don't have such experts on board, not anymore.  So we must do
it on our own somehow.

The mapping of GB18030 to Unicode is taken from glibc, see
etc/charsets/GB180302.map and etc/charsets/GB180304.map.  It is
possible that you are talking about a newer version of the GB18030
standard than these two mappings.  It is also possible that glibc has
since updated the mappings, and we failed to follow suit.  If so, we
need either to update the existing mappings or to add newer mappings.
Could you please see what needs to be done in this regard?