[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [bug-gnu-libiconv] cp936, cp950, cp1252, etc. does not behave like t
Mingye Wang (Arthur2e5)
Re: [bug-gnu-libiconv] cp936, cp950, cp1252, etc. does not behave like their windows counterparts
Thu, 24 Nov 2016 16:33:22 -0500
Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2
Bruno Haible wrote:
There are two implementations of 'iconv' in GNU, one in glibc, and one
in libiconv. Here you are writing about glibc behaviour, for which you
can report bugs in the glibc bugzilla. But I can give you some background
Hmm. I should find some time to forward some parts of this report (like
cp936) to glibc.
You can see from http://haible.de/bruno/charsets/conversion-tables/sources.html
that the site considers Windows versions up to October 2016.
Great collection... again.
libiconv includes the '0x80 U+20AC' mapping for CP936; glibc doesn't.
Maybe because this Euro sign is not contained in the GBK 1.0 standard
(see https://en.wikipedia.org/wiki/GBK#GBK_1.0). Maybe because U+20AC
is mapped to a different codepoint in GB18030 and GB18030 is meant to
be an extension of GBK.
I guess these tables should be kept 'like Windows' as long as they are
referring to a Windows Code Page. It seems that glibc simply did an
Also, confirmed working in Cygwin where `iconv` actually comes from GNU
cp950 has no mappings for HKSCS
It is not a good idea to propagate arbitrary modifications of existing
because it causes interoperability problems. You are actually calling it a
I am calling it a hack because MS is pushing a separate code page number
(951) to mask 950 in their Windows XP support package. Not quite the
case for later Windows releases...
As you can see from http://haible.de/bruno/charsets/conversion-tables/Big5.html
(search for windows-2016/CP950.TXT), you can see that on Windows 10, CP950
does *not* contain HKSCS extension mappings.
It seems that libiconv *does* have private area mappings for Big5's
user-defined blocks in cp950. glibc aliased CP950 to Big5, so it's going
to be their fault. Another false alarm.
On cygwin iconv seems to be able to accept \x87\x40 and give \ue000 but
not the other way around.
Likewise for the official mapping tables provided by Microsoft:
*todo: bug glibc for cp950 EUDC too*
Since it's a bidirectional
conversion, this assignment is not part of "best fit" behavior per .
You're mistaken. The point of the "best fit" converters in Microsoft is that
they document also the conversions that go only in one direction. i.e. that
I thought these round-trip parts are not doing "best fit" and should be
considered (somehow) normative.
0x81 and 0x8d for cp1252, etc.
See the tables provided by Microsoft:
As you can see, 0x81 and 0x8D are not mapped.
This actually brings up why I keep going straight to the round-trip
subset of these "best fit" mappings...
MS used to serve its cp950 mapping at their "go global" site, which
now redirects to a "not found" page at. This page points you to the
best fit mappings that, in turn, looks like what I have on current
versions of Windows. As a result, I actually thought these "WINDOWS"
mappings were somehow obsolete.
Yes the Windows converter does it differently...
In summary, choosing the right conversion table is a tricky choice. Don't
think that what a converter on Windows does it always the right or best option!
I still expect Windows Code Pages to be defined by Windows itself...
Description: OpenPGP digital signature