Re: [RFC 0/6] RFC: hw/display/virtio-gpu: problems with coloured cursors

[Gerd Hoffmann]

  Hi,

>    The cursor data virtio-gpu is receiving from the guest has
>    had the alpha channel pre-multiplied into the RGB components.

The kernel driver simply passes through whatever it gets from userspace.

Not sure what userspace passes to the kernel, I suspect it is whatever
typical GPUs can use unmodified as cursor sprite.

>    The cursor data virtio-gpu is receiving from the guest appears
>    to be in BGRA8888 format, while GTK/VNC both expect the data to
>    be in RGBA8888 format.

The format used by virtio-gpu is DRM_FORMAT_ARGB8888
(VIRTIO_GPU_FORMAT_B8G8R8A8_UNORM) on little endian guests.  Byteswapped
(DRM_FORMAT_BGRA8888 / VIRTIO_GPU_FORMAT_A8R8G8B8_UNORM) on bigendian
guests.

> This series has patches to virtio-gpu which reverse the RGB
> components and un-multiply the alpha channel.

I'd tend to simply pass the cursor format information to the ui and
leave it to the ui to convert if needed, or just use the data as-is if
possible (like SDL does).  Same approach we are using for the
framebuffer data.

IIRC the cursor code predates the adoption of pixman in qemu, maybe it
makes sense to redesign this around pixman images.

> I'm unclear whether its reference to "ARGB" here implies that
> seeing BGRA8888 data is intentional, or a guest kernel bug ?

Intentional.

> The spec says nothing at all about alpha pre-multiplication.
> I kind of think this is more likely to be a guest kernel bug,
> but its possible the spec just forgot to mention this ?

Not sure, missing in the spec I'd guess.

> Meanwhile I've absolutely no clue what impact endianness will
> have on this mess. All my testing thus far has been x86_64
> host (QEMU git HEAD) with x86_64 guest (Fedora 41).

Looking at virtio_gpu_update_cursor_data() my guess would be it's
broken (or working by pure luck and bug compatibility).  The
function goes lookup the resource containing the cursor data,
checks size, but doesn't even look at the format.

HTH & take care,
  Gerd