Re: about strings, symbols and chars.

[Jim Blandy]

In August 1999 I set up a plan to add multilingual text support to
Guile.  The plan was intended to allow Guile to arbitrarily mix text
from different languages in strings.

In general, C code needs to be able to directly inspect string
contents.  It's difficult to imagine any functional interface
providing one-character-at-a-time access efficiently enough to be
practical.  C code wants, and needs, direct access to string contents.
So whatever representation Guile uses for multi-lingual text will be
revealed by the C API --- we can't conceal it from C code.

There are two possible interesting encodings:
- UTF-8.  This is pretty much what the world is settling on.  GTK uses
  it; XML uses it; and so on.
- The MULE (MUlti-Lingual Emacs) encoding.  One of Stallman's primary
  goals for Guile is to replace the Emacs Lisp interpreter, and I
  *don't* think that Emacs buffers and strings should use different
  encodings.  In MULE, each character specifies a character set, and
  then an index into that character set.

But the really fascinating thing is, you don't actually need to
choose.  Both these encodings have all the nice properties you need to
manipulate them easily in C code:

- You can use them both in null-terminated strings.  strcpy and strlen
  work.
- You can use strchr and memchr to scan them for ASCII characters.
- You can use strstr to search them for arbitrary substrings, even
  substrings containing multi-byte characters.
- You never need to maintain "state" while scanning a string; you can
  always find character boundaries in finite time, just given a
  pointer into the middle of a string.
- You can tell how many bytes a character's encoding takes by looking
  at the first byte alone.

So, if you've got C code that works for one encoding, you can almost
always turn it into C code that works for the other just by dropping
in new lookup tables and constants.  This is a happy coincidence.

As far as the SCM_STRING_CHARS macro is concerned, this means that
eventually it will be providing access to the raw encoding.  I've
found it more convenient to use `unsigned char' in code which works
with encodings.

Since we don't have any real concept of encoding in Guile at the
moment, I don't think it's appropriate to specify that strings contain
Latin-1 characters or some such --- it's just whatever bytes that came
in on the port.  Yes, this is looser than one might want, but to solve
the problem properly requires some framework like the above applied
throughout Guile.