Re: Using libunistring for string comparisons et al

[Ludovic Courtès]

Hi,

Mike Gran <address@hidden> writes:

>> From:Ludovic Courtès <address@hidden>
>
>> > I know of two categories of bugs.  One has to do with case conversions
>> > and case-insensitive comparisons, which must be done on entire strings
>> > but are currently done for each character.  Here are some examples:
>> >
>> >   (string-upcase "Straße")         => "STRAßE"  
>> (should be "STRASSE")
>> >   (string-downcase "ΧΑΟΣΣ")        => "χαοσσ"  
>> (should be "χαoσς")
>> >   (string-downcase "ΧΑΟΣ Σ")       => "χαοσ σ"  
>> (should be "χαoς σ")
>> >   (string-ci=? "Straße" "Strasse") => #f        
>> (should be #t)
>> >   (string-ci=? "ΧΑΟΣ" "χαoσ")      => #f        
>> (should be #t)
>> 
>> (Mike pointed out that SRFI-13 does not consider these bugs, but that’s
>> linguistically wrong so I’d consider it a bug.  Note that all these
>> functions are ‘linguistically buggy’ anyway since they don’t have a
>> locale argument, which breaks with Turkish ‘İ’.)
>> 
>> Can we first check what would need to be done to fix this in 2.0.x?
>> 
>> At first glance:
>> 
>>   - “Straße” is normally stored as a Latin1 string, so it would need to
>>     be converted to UTF-* before it can be passed to one of the
>>     unicase.h functions.  *Or*, we could check with bug-libunistring
>>     what it would take to add Latin1 string case mapping functions.
>> 
>>     Interestingly, ‘ß’ is the only Latin1 character that doesn’t have a
>>     one-to-one case mapping.  All other Latin1 strings can be handled by
>>     iterating over characters, as is currently done.
>
> There is the micro sign, which, when case folded, becomes a Greek mu.
> It is still a single character, but, it is the only latin-1 character that,
> when folded, becomes a non-Latin-1 character

Blech.

It would have worked better with narrow == ASCII instead of
narrow == Latin1.  It’s a change we can still make, I think.

>>   - Case insensitive comparison is more difficult, as you already
>>     pointed out.  To do it right we’d probably need to convert Latin1
>>     strings to UTF-32 and then pass it to u32_casecmp.  We don’t have to
>>     do the conversion every time, though: we could just change Latin1
>>     strings in-place so they now point to a wide stringbuf upon the
>>     first ‘string-ci=’.
>> 
>> Thoughts?
>
> What about the srfi-13 case insensitive comparisons (the ones that don't
> terminate in question marks, like string-ci<)?  Should they remain
> as srfi-13 suggests, or should they remain similar in behavior
> to the question-mark-terminated comparisons?

Well, if maintaining two string comparison algorithms is reasonable,
then we can keep both; otherwise, I’d vote for the R6RS way.

> Mark is right that fixing this will not be pretty.  The case insensitive
> string comparisons, for example, could be patched like the attached
> snippet. If you don't find it too ugly of an approach, I could work on
> a real patch.

Indeed it’s quite inelegant.  ;-)

How about changing to narrow == ASCII and then string comparison would
be:

  if (narrow (s1) != narrow (s2))
    {
      /* Handle ß -> ss.  */
      if (!narrow (s1))
        widify (s1);
      else
        widify (s2);
    }

  if (narrow (s1))
    /* S1 and S2 are ASCII.  */
    return strcmp (char_data (s1), char_data (s2));
  else
    /* S1 and S2 are UTF-32.  */
    return u32_cmp (wide_char_data (s1), wide_char_data (s2));

Looks like that would remain reasonable while actually fixing our
problems.

As a side-effect, though, scm_from_latin1_locale would become slightly
less efficient because it’d need to check for non-ASCII chars.

Thanks,
Ludo’.