How should applications use direct-color mode?

[Tim Allen]

Kakoune[1] is an ncurses-based text editor that supports
syntax-highlighting themes, and therefore wants to display a wide variety
of colours in the terminal. Currently[2], for terminals where COLORS > 16
and can_change_color() is true, it will allocate the required colours with
init_color(), and otherwise it will try to pick the closest match from the
first COLORS entries in a copy of the xterm default 256-colour palette.

Now that ncurses 6.1 has been released with support for direct-colour
terminals, I'm investigating what would be required to teach Kakoune
(or any application, really) how to use it.

As I understand it, the first change is that the application needs to
be updated to call init_extended_pair() instead of init_pair(). This
function is only present when NCURSES_EXT_FUNCS is defined, so the
application can use #ifdef to choose which function is called.

The second change is that instead of passing palette entries to
init_extended_pair(), the application should pass actual packed RGB
colours, such as 0xFF7F00 for orange. However, curs_color(3X) says
"color values are expected to be in the range 0 to COLORS-1, inclusive
(including 0 and COLORS-1)", and at least with the ncurses 6.1+20180210-1
library from Debian Experimental "TERM=xterm-direct tput colors" prints
32767, or 0x7FFF. This implies that direct-color mode can only display
shades of blue, teal and green, and indeed that's the behaviour of
ncurses demos like "dots.c". Is it intended that COLORS=32767 be
interpreted as a sentinel value meaning "unlimited" rather than being
a specific limit?

The big question of course is: how can an application determine whether
the terminal suppors direct colour or palettized colour? I can't seem
to find any specific advice in curs_color(3x), and poking through the
ncurses source-code suggests that the relevent terminfo field ("RGB") is
used to avoid setting up a colour table (in start_color()) and in the
implementation of extended_color_content() but nowhere else.

Thus I assume that an application that wants to support direct-color
with ncurses 6.1 should do something like this:

  - Configure the application to use init_extended_pair() instead of
    init_pair() when available.
  - At startup, if the "RGB" terminfo field exists, interpret it according
    to the rules in user_caps(5) and record that we're in direct-color
    mode.
  - When the application wants to use a particular colour:
      - if we're in direct-color mode, encode the requested colour
        according to the rules described by the "RGB" terminfo field.
      - if can_change_color() and we've previously allocated a palette
        entry for this colour, use that.
      - if can_change_color(), and we haven't previously allocated a
        palette entry, allocate one and set it to the requested value
        with init_color().
      - otherwise, pick the closest colour according to what we think
        the palette might be.

Is that reasonable? Am I missing anything?

[1]: http://kakoune.org/
[2]: https://github.com/mawww/kakoune/blob/master/src/ncurses_ui.cc#L151