Re: [PATCH 2/5] [mingw]: Have compiled-file-name produce valid names.

[Noah Lavine]

Hello all,

I have another issue to raise. I think this is actually parallel to
some of the stuff in the (web) module, as you will see.

I've always thought it was ridiculous and hackish that I had to escape
spaces in path strings. For instance, I have a folder called "Getting
a Job" on my desktop, whose path is ~/Desktop/Getting\ a\ Job.

The reason this strangeness enters is that path strings are actually
lists (or vectors) encoded as strings. Conceptually, the path
~/Desktop/Getting\ a\ Job is the list ("~" "Desktop" "Getting a Job").
In this representation, there are no escapes and no separators. It
always seemed cleaner to me to think about it that way. I think there
should be some mechanism by which Guile users never have to think
about escaping spaces (and any other characters they want in their
paths). We don't have to represent them with lists or vectors, but
there should be some mechanism for avoiding this.

I said this is similar to the (web) module because of all of the
discussion there of how HTTP encodes data types in text, and how it's
better to think of a URI as URI type rather than a special string,
etc. I think the same issue applies here - you've got list (or a list
of lists, if you have a whole command-line with arguments) encoded as
a string using ' ' and '/' as separators, and then you have to escape
those characters when you want to use them in a different way, and the
whole thing gets unnecessarily complicated because the right way to
think about this is as lists of strings.

Noah

On Tue, May 3, 2011 at 11:59 PM, Mark H Weaver <address@hidden> wrote:
> Andy Wingo <address@hidden> writes:
>> That's the crazy thing: file names on GNU aren't in any encoding!  They
>> are byte strings that may or may not decode to a string, given some
>> encoding.  Granted, they're mostly UTF-8 these days, but users have the
>> darndest files...
> [...]
>> On Tue 03 May 2011 00:18, address@hidden (Ludovic Courtès) writes:
>>> I think GLib and the like expect UTF-8 as the file name encoding and
>>> complain otherwise, so UTF-8 might be a better default than locale
>>> encoding (and it’s certainly wiser to be locale-independent.)
>>
>> It's more complicated than that.  Here's the old interface that they
>> used, which attempted to treat paths as utf-8:
>>
>>   http://developer.gnome.org/glib/unstable/glib-Character-Set-Conversion.html
>>   (search for "file name encoding")
>>
>> The new API is abstract, so it allows operations like "get-display-name"
>> and "get-bytes":
>>
>>   http://developer.gnome.org/gio/2.28/GFile.html  (search for "encoding"
>>   in that page)
>>
>>   "All GFiles have a basename (get with g_file_get_basename()). These
>>   names are byte strings that are used to identify the file on the
>>   filesystem (relative to its parent directory) and there is no
>>   guarantees that they have any particular charset encoding or even make
>>   any sense at all. If you want to use filenames in a user interface you
>>   should use the display name that you can get by requesting the
>>   G_FILE_ATTRIBUTE_STANDARD_DISPLAY_NAME attribute with
>>   g_file_query_info(). This is guaranteed to be in utf8 and can be used
>>   in a user interface. But always store the real basename or the GFile
>>   to use to actually access the file, because there is no way to go from
>>   a display name to the actual name."
>
> In my opinion, this is a bad approach to take in Guile.  When developers
> are careful to robustly handle filenames with invalid encoding, it will
> lead to overly complex code.  More often, when developers write more
> straightforward code, it will lead to code that works most of the time
> but fails badly when confronted with weird filenames.  This is the same
> type of problem that plagues Bourne shell scripts.  Let's please not go
> down that road.
>
> There is a better way.  We can do a variant of what Python 3 does,
> described in PEP 383 <http://www.python.org/dev/peps/pep-0383/>.
>
> Basically, the idea is to provide alternative versions of
> scm_{to,from}_stringn that allow arbitrary bytevectors to be turned into
> strings and back again without any lossage.  These alternative versions
> would be used for operations involving filenames et al, and should
> probably also be made available to users.
>
> Basically the idea is that "invalid bytes" are mapped to code points
> that will never appear in any valid encoding.  PEP 383 maps such bytes
> to a range of surrogate code points that are reserved for use in UTF-16
> surrogate pairs, and are otherwise considered invalid by Unicode.  There
> are other possible mapping schemes as well.  See section 3.7 of Unicode
> Technical Report #36 <http://www.unicode.org/reports/tr36/> for more
> discussion on this.
>
> I can understand why some say that filenames in GNU are not really
> strings but rather bytevectors.  I respectfully disagree.  Filenames,
> environment variables, command-line arguments, etc, are _conceptually_
> strings.  Let's not muddle that concept just because the transition to
> Unicode has not yet been completed in the GNU world.
>
> Hopefully in the future, these old-style POSIX byte strings will once
> again become true strings in concept.  All that's required for this to
> happen is for popular software to agree to standardize on the use of
> UTF-8 for all of these things.  This is reasonably likely to happen at
> some point.
>
> In practice, I see no advantage to calling them bytevectors other than
> to allow lossless storage of oddball filenames.  It's not as if any sane
> user interface is going to display them in hex.  Think about it.  What
> are you really going to do with the bytevector version, other than to
> store it in case you want to convert it back into a filename,
> environment variable, or command-line argument?  Think about the mess
> that this will make to otherwise simple code.  Also think about the
> obscure bugs that will arise from programmers who balk at this and
> simply pass around the strings instead.
>
> Let's keep things simple.  Let's use plain strings for everything that
> is _conceptually_ a string.  Let's instead deal with the occasional
> ill-encoded-filename by allowing strings to represent these oddballs.
>
>     Best,
>      Mark
>
>