[Qemu-devel] Discussion: vnc: memory leak in zrle_compress_data

[fangying]

Hi Gerd,

Memory leak is observed in zrle_compress_data when we are doing some 
AddressSanitizer tests. The leak stack is as bellow:

=================================================================
==47887==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 29760 byte(s) in 5 object(s) allocated from:
    #0 0xffffa67ef3c3 in __interceptor_calloc (/lib64/libasan.so.4+0xd33c3)
    #1 0xffffa65071cb in g_malloc0 (/lib64/libglib-2.0.so.0+0x571cb)
    #2 0xffffa5e968f7 in deflateInit2_ (/lib64/libz.so.1+0x78f7)
    #3 0xaaaacec58613 in zrle_compress_data ui/vnc-enc-zrle.c:87
    #4 0xaaaacec58613 in zrle_send_framebuffer_update ui/vnc-enc-zrle.c:344
    #5 0xaaaacec34e77 in vnc_send_framebuffer_update ui/vnc.c:919
    #6 0xaaaacec5e023 in vnc_worker_thread_loop ui/vnc-jobs.c:271
    #7 0xaaaacec5e5e7 in vnc_worker_thread ui/vnc-jobs.c:340
    #8 0xaaaacee4d3c3 in qemu_thread_start util/qemu-thread-posix.c:502
    #9 0xffffa544e8bb in start_thread (/lib64/libpthread.so.0+0x78bb)
    #10 0xffffa53965cb in thread_start (/lib64/libc.so.6+0xd55cb)

This leak stack can be easily reproduced in the case that a client repeatedly 
does vnc connect/disconnect .

To get the leak stack, we can compile qemu with
--extra-ldflags=-Wl,--build-id -Wl,-z,relro -Wl,-z,now -lasan'
'--extra-cflags=-O0 -g -fno-omit-frame-pointer -fno-stack-protector 
-fsanitize=address -fsanitize=leak' using gcc that supports asan.

It is obvious that there may be memory leak in the zlib/zrle compress stuff.
IIUC, *deflateInit2* is called with the local VncState vs->zrle.stream when a 
client is connected the vnc server. And then *deflate* is called to do the 
encoding. Finally *deflateEnd* is called in vnc_zrle_clear when a connection is
closed.

I had not think it out why this memory leak could happen here.
It is noticed that deflateInit2 is called with the local vs,
however deflateEnd is called with the origin vs.
The local vs is copied to the origin vs in vnc_async_encoding_start and 
vnc_async_encoding_end on the contrary.

Have you got any idea on this issue ?

Thanks.
Ying Fang