Re: Unicode and Guile

[Andy Wingo]

On Tue, 11 Nov 2003, Tom Lord wrote:

> Thanks for the pointer to the Python type (on which I won't comment
> :-).   Thanks for the excuse to think about this more.

And thanks for thinking this through a lot more properly than I was, and
for caring about the problem, and for having patience with the ignorant
:-)

> ** CHAR? Makes No Sense In Unicode

I think I'm starting to get a clue. Case mapping demonstrates this
pretty clearly... Incidentally, GLib's function for this is evidently
broken:

gunichar    g_unichar_toupper               (gunichar c);

Although they do have g_utf8_strup, which operates on a string and does
the correct thing.

> * The Proposal
> 
>   The proposal has two parts.   Part 1 introduces a new type, TEXT?, 
>   which is a string-like type that is compatible with Unicode, and
>   a subtype of TEXT?, GRAPHEME?, to represent "conceptual
>   characters". 

Wow, you really have thought a lot more about this than I have.

>   It is important to note that, in general, EQV? and EQUAL?  do _not_
>   test for grapheme equality.  GRAPHEME=? must be used instead.

I can see why EQV? shouldn't test for equality: a precomposed grapheme
can be the same as one made with combining characters. But why not
overload EQUAL?, given that they would display the same (with a suitable
glyph rendering library)? Perhaps this is not possible in portable
Scheme? If this question is ignorant, my apologies.

>   So, texts really need markers that work like those in Emacs:

It does indeed appear so. I withdraw my ridicule of this idea :-P

> * Optional Changes to CHAR? and STRING?
> 
>     ~ TEXT? values contain an "encoding" attribute, just as strings
>       do (utf-8, etc.)

Why should an implementation support more than one encoding, internally?

>     ~ (string? a-text-value) => #t

Would be difficult with Guile, given the C interface... Perhaps if there
were an abstract string type, with "simple strings" as a subtype, then C
functions wanting a string (just for reading) would not call
SCM_STRING_CHARS but scm_string_chars, or the like...

> [I]f I'm sitting in california and write a protable Scheme program
> that generates anagrams of a name, it'd be awefully swell if (a) My
> code doesn't have to "know" anything special about unicode internals;
> (b) my code works when passed her name as input.

Indeed.


Overall, your proposal is IMHO well-thought out, and is of high quality.
I am humbled :). I hope something like this can go into Guile soon.

Cheers,

wingo.