Re: scm_to_locale_stringbuf

[Mike Gran]

> From: Neil Jerram address@hidden

> I'm afraid I don't understand the problem, on two counts.
> 
> 1. The doc (in the manual) says that scm_to_locale_stringbuf doesn't
> add a terminating \0.  So presumably any \0s present must be padding.
> 
> 2. The doc also says that if scm_to_locale_stringbuf's return value
> is > max_len (as it would be in your case), the caller should call it
> again with a larger buffer.
> 

Right now, the internal coding of strings is an unspecified 8-bit encoding, and 
is assumed to be compatible with the locale in which it is being run.

So if I have a guile string with some 8-bit character that is between 128 and 
255, it just gets passed through.  If I request the contents of that string 
from C with scm_to_locale_string, it just returns the buffer of the scheme 
string.

But, in future, scm_to_locale_string or scm_to_locale_stringbuf should actually 
do the proper conversion to the current locale so that wide characters are 
printed properly.

So, if we move the internal representation of strings away
from unspecified 8-bit data and toward something concrete,
like ISO-8859-1 or UCS-4, and if a program is running in an
environment where a locale that has a multibyte encoding
like UTF-8, then the created locale string could have multi-byte characters.

Consider a scheme string that is internally the single
character "LATIN SMALL LETTER A WITH ACUTE", which is
U+00E1.  If the locale were some sort of UTF-8, like
en_US.utf-8, this letter should become the two bytes 0xC3
and 0xA1 when converted to the locale.

So what should happen in this case if I call
scm_to_locale_stringbuf (str, buf, 1)?  Note that here BUF
can only contain 1 byte.  Should the one byte 0xC3 be
copied into it, which creates an illegal string?  Or,
should nothing be copied into it.  In either case, there
should be some mechanism in the API to provide information
that an incomplete last character has occurred, because
outputting just the one byte 0xC3 would cause problems
somewhere down the road.

So what I was saying was that in this case maybe the best
thing to do would be to pad the output buffer with '\0'
instead of putting in half of a multibyte character, and
then signal that there is some padding at the end of the
string.

For instance, one could have a function
scm_to_locale_stringbufn (SCM str, char *buf, size_t max_len, size_t *len_used)
where LEN_USED is size of the buffer that was actually
used.

Sorry for the book-length explanation,

Mike Gran