Re: [vile] problem with 'wide characters' (utf-8) under macosx

[j. van den hoff]

On Sat, 06 Dec 2014 15:15:26 +0100, Thomas Dickey <address@hidden> wrote:

> On Sat, Dec 06, 2014 at 02:42:11PM +0100, j. van den hoff wrote:
> > On Sat, 06 Dec 2014 13:37:40 +0100, Thomas Dickey <address@hidden>  
> > wrote:
> ...
> > >With "auto", vile inspects the file as it reads it, and decides what  
> > its
> > >encoding is.  One problem with that approach is if you start with a  
> > plain
> > >ASCII file and want to change it to UTF-8.  You can do that by doing
> > >
> > >    :setl file-encoding=utf-8
> > >
> > >before introducing the non-ASCII text.  (The "setl" is needed because
> > >vile maintains that value as a local-setting for each buffer, and a  
> > plain
> > >"set" will not tell it to revise its idea of a particular buffer).
> >
> > I see.  thanks.  the question still remains, why `file-encoding=auto'  
> > works
> > (as it is intended, I understand) when opening the respective input file
> > (already containing utf-8 characters) in urxvt while it does not when  
> > using
> > the Terminal.app.  it is not important any more for me (I'll now stick  
> > with
> > the default setting of file-encoding') but still...
>
> I'd suspect that my shell environment (such as locale) in Terminal.app
> was different from the one in X11.

I thought `auto' would guess the encoding just from the file content?
and the locale settings are as indicated previously (LC_CTYPE=de_DE.utf-8  
(x11) and `utf-8' (Terminal.app), respectively)
and character encoding of Terminal.app consequently set to "Unicode  
(UTF-8)".

> On my Mac's, I setup one user to own the gui, and a different user who
> ssh's to various places (and most of the editing that I do is via the
> latter), so I may be overlooking some case where X11 and Terminal.app
> environments disagree.
>
> > I see also some other quirk:  when actually entering further utf-8 sign  
> > (say
> > german diaeresis) to the file within Terminal.app (where now the already
> > existing chars are displayed correctly), those are displayed as `??'  
> > (two
>
> vile will display a non-UTF-8 byte with a single "?" mark followed by
> its hexadecimal value.  But I don't recall (or see in the code) something
> that would do a double "??".  You could (I think...) get that from some
> obscure translating problem between UTF-8 and one of the ISO-8859-x's
> that are used in the "8bit" encoding, but I'm not sure where to look :-(
>
> If I had something like that which I could easily reproduce, the only
> way I know to track it down (as a bug report) would be to build vile with
> the debugging trace, and see who did it.

I played around with this a bit more and it turns out that a simple  
display redraw
suffices to get the correct display, i.e. the two `??' seem spuriously  
created by terminal.app
and go away when leaving
input mode and hitting crtl-l. this sure looks like some issue with  
Terminal.app
(but at least vim does not trigger this behaviour and immediately displays  
the
characters correctly).

so if you have access to a mac (and presuming you use an utf-8 locale  
yourself)
you should be able to reproduce it:
what happens if you edit a new file in terminal.app and input, e.g.,  
`alt-u o' (i.e.
first hold down `alt' and press `u' then release and press `o'). this is  
the
key-combo for creating the letter `ö'. when I do this I get what I  
described:
two `??' are displayed until I leave input mode and perform undo/redo  
(hittig `uu')
or do a redraw (ctrl-L)
at which point the letter is displayed correctly. do you see something  
different?

I now also recall that by and then when scrolling or similar I have seen  
spurious `?'
in vile when using terminal.app, which, too, went away after crtl-L.
does this give you a better idea, whether this might be caused by the way
vile manages the display? as I said it might be a terminal.app bug but if  
so, it seems
not be triggered by vim at least (which I understand manages the screen  
not via
ncurses IIRC?).

thx/joerg

> > question marks), until I leave display of that buffer (by going to an
> > alternate file) and coming back at which time the symbol appears  
> > correctly
> > displayed (or by leaving input mode and performing undo/redo of the  
> > edit,
> > which also produces correct display of the character).  any idea how  
> > _this_
> > behaviour comes about (again:  this does not happen when using `vim'  
> > instead,
> > so it seems not to be just a problem of the terminal.app)?


--
Using Opera's revolutionary email client: http://www.opera.com/mail/