bug#69598: 29.2; colour support based on $TERM value not terminfo database

[chohag]

Eli Zaretskii writes:
> > From: chohag@jtan.com
> > cc: chohag@jtan.com, 69598@debbugs.gnu.org
> > Comments: In-reply-to Eli Zaretskii <eliz@gnu.org>
> >    message dated "Fri, 08 Mar 2024 14:22:52 +0200."
> > Date: Fri, 08 Mar 2024 14:23:26 +0000
> > 
> > > > If we're at the point where the terminal's name matters one jot,
> > > > that is to say when the value of $TERM is used for _anything_ other
> > > > than selecting a terminfo entry, that is a bug.
> > >
> > > It is not necessarily a bug.  Emacs needs to be able to convert
> > 
> > It's a problem inasmuch as it is a violation of the terminfo protocol
> > and takes a sharp turn into non-standards territory. Perhaps bug
> > is not the best term. It's certainly a red flag.
>
> We only do that for handling the colors, and I'm not sure I can see
> how terminfo can help us here.  terminfo knows how many colors a
> terminal supports and which escape sequences to use for that, but it
> doesn't know which colors will be produced, in terms of the RGB
> triplets.  So Emacs needs something else to support colors on text

Terminfo doesn't ``know'' what a terminal supports. Terminfo is the
database and routines to look up capabality values. It's the
applications which use terminfo that ``know'' what those capabilities
do and, of course, many of them have a conventional use as described
in terminfo(5).

In order to describe capabilities of terminals which hadn't been
made in 1980 it can be expanded with new capabilities. For example
setb24 and setf24 are terminfo extensions which as near as I can
tell originated in emacs to do exactly what you describe.

https://www.gnu.org/software/emacs/manual/html_node/efaq/Colors-on-a-TTY.html
mentions setb24 and setf24. In fact it's the only page on the web
that I can find (apart those linking to it) with any record of these
controls:

        Emacs 26.1 and later support direct color mode in terminals.
        If Emacs finds Terminfo capabilities `setb24' and `setf24',
        24-bit direct color mode is used. The capability strings
        are expected to take one 24-bit pixel value as argument and
        transform the pixel to a string that can be used to send
        24-bit colors to the terminal.

        Standard terminal definitions don't support these capabilities
        and therefore custom definition is needed.

          setb24=\E[48\:2\:\:%p1%{65536}%/%d\:%p1%{256}%/%{255}%&\
             %d\:%p1%{255}%&%dm,
          setf24=\E[38\:2\:\:%p1%{65536}%/%d\:%p1%{256}%/%{255}%&\
             %d\:%p1%{255}%&%dm,

``24-bit direct color mode is use'' with no further qualifications.
It doesn't say that emacs also needs to have special knowledge about
the terminal's model.

The page continues with examples for further developments but in
every case implies or outright says that certain capabilities are
the only requirement, not specific models.

> terminals, and that something comes from the lisp/term/ files.  Those
> files also set up Emacs for various non-standard function keys
> produced by the terminal, and support other features, like copy/paste
> etc.
>
> So these files are not a bug, they are the foundation of several
> important features in Emacs.

Their overriding what terminfo tells it is the buggy part. Terminfo
says ``terminal foo has RGB or setf24 and setb24 capabilities and
supports 16M colours'' but emacs disagrees.

Incidentally, both of the examples you gave are already supported
in some fashion by terminfo capabilities, for example:

        The kNNN capabilities map non-standard keys.

        bracketed+paste|xterm bracketed paste,
                BD=\E[?2004l,
                BE=\E[?2004h,
                PE=\E[201~,
                PS=\E[200~,

> > > X-style color names to escape sequences it should send to the terminal
> > > to produce a similar color.  That conversion is specific to the
> > > terminal, so Emacs relies on the name of the terminal to set up the
> > > conversion.
> > 
> > Hmm. If only there was some sort of database that described
> > terminal-specific capabilities and features and how to use them.
> > It could also ensure the programs were portable, even to terminals
> > that haven't been created yet!
>
> Yes.  But AFAIK there's no such database, so Emacs must use its own
> database.

I'm sorry, I was trying to use humour to point out that terminfo
is that database.

> So you now agree with me that a terminal file in lisp/term is needed
> for this situation?  Or are there issues that are still unaccounted
> for, as far as color support is concerned?

I don't agree. A new lisp/term file achieved the desired result,
or rather the appearance of it, but for all of the wrong reasons:
because the terminal lied about what it was.

Looking at it another way if a user uses one of the terminals emacs
supports but with certain features enabled or disabled by a custom
terminfo entry there's nothing to say that the name of that entry
has to follow any particular convention. A site prefix is a well
established naming scheme so something like mycompany-footerm does
not seem like an unreasonable name and nothing formally or informally
in terminfo or termcap, or in emacs' documentation for that matter,
suggests that it wouldn't work.

Indeed I'm quite surprised to find this behaviour: emacs, running
in xterm and informed it is running on a terminal identical to
xterm, does not act the same as it does when that description comes
under the *label* xterm. That's like the name of a variable influencing
its contents.

In short, if emacs only supports colour (or any other feature) on
specific terminals it knows about in advance, it should say so and
not imply that a terminfo or termcap capability is sufficient.

As it stands this paragraph:

        If you think that your terminal supports colors, but Emacs
        won't use them, check the termcap entry for your display
        type for color-related capabilities.

... should be amended to say ``check if your terminal or emulator is
one of these supported models: <list>'' but couldn't emacs believe
and use what terminfo has told it, otherwise why did it ask?

Matthew