Re: [Groff] bullets render as question marks

[Ingo Schwarze]

Hi Aaron,

Aaron Davies wrote on Mon, Nov 30, 2015 at 08:00:06PM -0500:

> $ locale charmap
> ANSI_X3.4-1968

Heh.  I didn't see that name for ASCII before and had to look it
up to learn what it means.  :)

The GNU nroff(1) script never heard about that name for ASCII
either, so it falls back to LC_ALL, and since that is "C",
it falls back further to $LESSCHARSET.  Is $LESSCHARSET defined
on your system, and if so, what is its value?

> $ echo '\(bu' | groff -Tascii | hexdump -C
> 00000000  2b 08 6f 0a 0a 0a 0a 0a  0a 0a 0a 0a 0a 0a 0a 0a  |+.o.............|
[...]
> $ echo '\(bu' | groff -mtty -Tascii | hexdump -C
> 00000000  2b 08 6f 0a 0a 0a 0a 0a  0a 0a 0a 0a 0a 0a 0a 0a  |+.o.............|
[...]

That's both correct.

> $ echo '\(bu' | groff -Tutf8 | hexdump -C
> 00000000  c2 b7 0a 0a 0a 0a 0a 0a  0a 0a 0a 0a 0a 0a 0a 0a  |................|
[...]
> $ echo '\(bu' | groff -mtty -Tutf8 | hexdump -C
> 00000000  c2 b7 0a 0a 0a 0a 0a 0a  0a 0a 0a 0a 0a 0a 0a 0a  |................|
[...]

 0xc2 - 0xc0 = 0x02; 0x0200 >> 2 = 0x80
 0xb7 - 0x80 = 0x37; 0x37 + 0x80 = 0xb7

So that is U+00B7, MIDDLE DOT.

Perhaps not the best possible choice, but not wrong either, and
certainly valid UTF-8.

> i think i can reconstruct the final rendering command now:
> 
> /usr/bin/iconv -f utf-8 -t utf-8 $TMPFILE|/usr/bin/groff -mtty-char \
> -P-c -mandoc -Tutf8|/usr/bin/iconv -f utf-8 -t ANSI_X3.4-1968//translit

That is ugly.  Someone is trying to sell you hardware?  Trying to
burn as many processor cycles as possible, and then some more?  ;-)

Apart from the obvious contortion, it doesn't even look right.  If the
input would really contain non-ASCII characters, the first iconv stage
would be insufficient, the pipeline would need preconv(1) before
groff.  

I suggest fixing your system to simply do this instead:

  /usr/bin/groff -P-c -mandoc -Tascii

That's all that is needed.  Even the -mtty-char is redundant,
your troffrc includes tty.tmac anyway.

> and the first two stages:
> 
> $ /usr/bin/iconv -f utf-8 -t utf-8 $TMPFILE | \
> /usr/bin/groff -mtty-char -P-c -mandoc -Tutf8|hexdump -C
[...]
> 000000a0  20 20 20 c2 b7 20 20 20  2d 08 2d 2d 08 2d 62 08  |   ..   -.--.-b.|
[...]
> 
> which renders as what i presume is supposed to be a unicode bullet
> (though it's a bit hard to tell in any of my terminal fonts):

MIDDLE DOT, not BULLET.

> though i can't figure out how 0xc2b7 corresponds to codepoint 2022

It doesn't, it's U+00B7, \(md, see the fallback in tty.tmac.

> the final stage, where iconv is supposed to convert it to ascii,
> is where the bullet turns into a question mark:

Yeah, so your's is an iconv(1) problem, not a groff(1) problem.
In this case, iconv(1) looks like a solution in search of a problem,
and like a broken solution, too.  :-/

Just get rid of the pointless iconv(1), and you are fine.

> so it looks like the real problem is that the groff stage in the
> middle of the pipeline isn't generating a unicode output that iconv
> understands how to transliterate

No, groff output is valid, i don't know why iconv(1) fails to handle it.

> incidentally, the manpage where i originally discovered this issue
> has a similar problem with \', but as it's using that to represent
> an apostrophe, which is wrong, that's not really your problem

True, that's an acute accent.

> (the odd thing is that groff is rendering that as 0xc2b4, which
> AFAICT *is* correct UTF-8, so maybe my iconv just doesn't have a
> transliteration rule for it?)

Yes, 0xc2b4 = U+00B4 = ACUTE ACCENT, that is fine.

Indeed, you need to debug iconv, not groff.

Yours,
  Ingo