Re: From wchar_t to char32_t

[Paul Eggert]

On 2023-07-03 15:00, Bruno Haible wrote:
>    Level 3: Behave correctly. Don't split a 2-Unicode-character sequence.
>             This is what code that uses mbrtoc32() does, when it has the
>             lines
>                  if (bytes == (size_t) -3)
>                    bytes = 0;
>             and uses !mbsinit (&state) in the loop termination condition.

With diffutils even level 3 would not suffice, since diffutils truncates 
at input byte boundaries, so it doesn't suffice to merely treat (size_t) 
-3 as zero even if one also checks mbsinit. Instead, one would have to 
treat all the characters in the sequence ABBB... (where A is an ordinary 
multibyte character and the Bs all return (size_t) -3) as a single unit, 
because one cannot truncate in the middle of that sequence. Or wait a 
minute - in theory I suppose it could even be an arbitrary sequence of 
As and Bs, so long as the total "sizes" of the As equals the number of 
bytes in the original byte sequence that stands for a series of characters.

The diffutils truncation approach also has problems with coding systems 
that have shift state, but that's OK: nobody uses these coding systems 
with GNU apps as they're not practical. Similarly, any platform where 
mbrtoc32 returns (size_t) -3 won't be practical with GNU apps, so it 
should be OK for diffutils to not worry about this possibility either, 
given that it would be a hassle to support it. We don't have time to 
support every oddball coding system that POSIX allows.