[RFC 0/6] RFC: hw/display/virtio-gpu: problems with coloured cursors

[Daniel P . Berrangé]

Help needed ! This is not for merge, just demo.

I started working on a simple bug in the GTK-VNC 'alpha cursor' encoding
support and in testing that uncovered what appear to be several bugs in
QEMU related to cursor handling across different components.

The two core behaviours are seen:

 * In some display backends (SDL, GTK), areas of the cursor
   with partial alpha are less saturated than they ought to be.

   The cursor data virtio-gpu is receiving from the guest has
   had the alpha channel pre-multiplied into the RGB components.

   This is not apparent in the VNC backend, since the VNC server
   is *supposed* to pre-multiply alpha but forgot to do so. Thus
   the virtio-gpu issue is hiding the VNC server issue.

 * In some display backends (GTK, VNC), the cursor RGB components
   are reversed.

   The cursor data virtio-gpu is receiving from the guest appears
   to be in BGRA8888 format, while GTK/VNC both expect the data to
   be in RGBA8888 format.

   This is not apparent in the SDL backend, since that is telling
   SDL to use BGRA8888 format when loading the cursor.

This series has patches to virtio-gpu which reverse the RGB
components and un-multiply the alpha channel.

I am not especially confident this is correct though.

The virtio spec is light on details:

  
https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/virtio-v1.1-cs01.html#x1-3200007

but says

   "In 2D mode the virtio-gpu device provides support
    for ARGB Hardware cursors and multiple scanouts
    (aka heads)."

I'm unclear whether its reference to "ARGB" here implies that
seeing BGRA8888 data is intentional, or a guest kernel bug ?

The spec says nothing at all about alpha pre-multiplication.
I kind of think this is more likely to be a guest kernel bug,
but its possible the spec just forgot to mention this ?

Meanwhile I've absolutely no clue what impact endianness will
have on this mess. All my testing thus far has been x86_64
host (QEMU git HEAD) with x86_64 guest (Fedora 41).

The problem isn't obvious since guests usually have monochrome
cursors by default. To aid in testing, on *both* your host and
your guest OS do:

  git clone https://gitlab.com/berrange/cursordemo
  cd cursordemo
  ./setup.sh
  gnome-tweaks

Navigate to Apperance, and select "RGB" or "Rainbow" as the
cursor theme.

Now compare the guest OS rendered cursor to what you see on
the host OS. They should obviously match 100% since they're
using the same theme.

Daniel P. Berrangé (6):
  ui: add more cursor helper methods
  hw/display/virtio-gpu.c: reverse alpha pre-multiplication
  hw/display/virtio-gpu: fix pixel ordering from BGRA8888 to RGBA8888
  ui/vnc: pre-multiply alpha with alpha cursor
  ui/sdl: load cursor in RGBA8888 format not BGRA8888
  ui: add ability to dump the raw cursor bytes

 hw/display/virtio-gpu.c |  3 ++
 include/ui/console.h    |  7 +++++
 ui/cursor.c             | 70 +++++++++++++++++++++++++++++++++++++++++
 ui/sdl2.c               |  2 +-
 ui/vnc.c                |  7 ++++-
 5 files changed, 87 insertions(+), 2 deletions(-)

-- 
2.47.1