[RFC 4/6] ui/vnc: pre-multiply alpha with alpha cursor

[Daniel P . Berrangé]

The RFB specification for the alpha cursor encoding requires that the
alpha channel is pre-multiplied into the RGB components. This worked
by luck previously since the virtio-gpu device was not reversing the
pre-multiplication on data received from the guest. Now virtio-gpu is
fixed, the VNC server must apply pre-multiplication itself.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 ui/vnc.c | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/ui/vnc.c b/ui/vnc.c
index 9241caaad9..5ffb50109d 100644
--- a/ui/vnc.c
+++ b/ui/vnc.c
@@ -996,6 +996,7 @@ static int vnc_cursor_define(VncState *vs)
     }
 
     if (vnc_has_feature(vs, VNC_FEATURE_ALPHA_CURSOR)) {
+        g_autoptr(QEMUCursor) tmpc = cursor_copy(c);
         vnc_lock_output(vs);
         vnc_write_u8(vs,  VNC_MSG_SERVER_FRAMEBUFFER_UPDATE);
         vnc_write_u8(vs,  0);  /*  padding     */
@@ -1003,7 +1004,11 @@ static int vnc_cursor_define(VncState *vs)
         vnc_framebuffer_update(vs, c->hot_x, c->hot_y, c->width, c->height,
                                VNC_ENCODING_ALPHA_CURSOR);
         vnc_write_s32(vs, VNC_ENCODING_RAW);
-        vnc_write(vs, c->data, c->width * c->height * 4);
+
+        // Alpha is required to be pre-multiplied into RGB components
+        cursor_multiply_alpha(tmpc);
+
+        vnc_write(vs, tmpc->data, c->width * c->height * 4);
         vnc_unlock_output(vs);
         return 0;
     }
-- 
2.47.1