[Patch] SRFI-13 string-tokenize is wrong

[Matthias Koeppe]

Hi,

the Guile implementation of SRFI-13 `string-tokenize' gets the meaning
of the `token-set' argument wrong.

Quoting the SRFI:

| string-tokenize s [token-set start end] -> list
| 
|     Split the string s into a list of substrings, where each substring
|     is a maximal non-empty contiguous sequence of characters from the
|     character set token-set.
| 
|         * token-set defaults to char-set:graphic (see SRFI 14 for more
|           on character sets and char-set:graphic).
| 
|     [...]    
| 
|     (string-tokenize "Help make programs run, run, RUN!") 
|     => ("Help" "make" "programs" "run," "run," "RUN!")
 
In Guile (1.5 branch):

      (string-tokenize "Help make programs run, run, RUN!") 
      => ("Help" "make" "programs" "run," "run," "RUN!")  ; OK

but:

      (string-tokenize "Help make programs run, run, RUN!" char-set:graphic)
      => (" " " " " " " " " ")  ; WRONG

The corresponding tests in srfi-13.test are also wrong.

I suggest fixing this bug in both the stable and the unstable branch,
so that incorrect uses of `string-tokenize' in user code are avoided.

The attached patch fixes the bug and also removes the Guile-specific
extension of `string-tokenize' to accept a character as the
`token-set' argument because it is inconsistent with both the
Guile-specific procedure documentation and with the correct behavior
of `string-tokenize' when a character set is passed as `token-set'.

-- 
Matthias Köppe -- http://www.math.uni-magdeburg.de/~mkoeppe

txt0rQbFdTh2I.txt
Description: Text Data