Re: [Qemu-devel] Using the qemu tracepoints with SystemTap

[Stefan Hajnoczi]

On Mon, Sep 12, 2011 at 4:33 PM, William Cohen <address@hidden> wrote:
> The RHEL-6 version of qemu-kvm makes the tracepoints available to SystemTap. 
> I have been working on useful examples for the SystemTap tracepoints in qemu. 
> There doesn't seem to be a great number of examples showing the utility of 
> the tracepoints in diagnosing problems. However, I came across the following 
> blog entry that had several examples:
>
> http://blog.vmsplice.net/2011/03/how-to-write-trace-analysis-scripts-for.html
>
> I reimplemented the VirtqueueRequestTracker example from the blog in 
> SystemTap (the attached virtqueueleaks.stp). I can run it on RHEL-6's 
> qemu-kvm-0.12.1.2-2.160.el6_1.8.x86_64 and get output like the following. It 
> outputs the pid and the address of the elem that leaked when the script is 
> stopped like the following:
>
> $ stap virtqueueleaks.stp
> ^C
>     pid     elem
>   19503  1c4af28
>   19503  1c56f88
>   19503  1c62fe8
>   19503  1c6f048
>   19503  1c7b0a8
>   19503  1c87108
>   19503  1c93168
> ...
>
> I am not that familiar with the internals of qemu. The script seems to 
> indicates qemu is leaking, but is that really the case?  If there are 
> resource leaks, what output would help debug those leaks? What enhancements 
> can be done to this script to provide more useful information?

Leak tracing always has corner cases :).

With virtio-blk this would indicate a leak because it uses a
request-response model where the guest initiates I/O and the host
responds.  A guest that cleanly shuts down before you exit your
SystemTap script should not leak requests for virtio-blk.

With virtio-net the guest actually hands the host receive buffers and
they are held until we can receive packets into them and return them
to the host.  We don't have a virtio_reset trace event, and due to
this we're not accounting for clean shutdown (the guest driver resets
the device to clear all virtqueues).

I am submitting a patch to add virtio_reset() tracing.  This will
allow the script to delete all elements belonging to this virtio
device.

> Are there other examples of qemu probing people would like to see?

The malloc/realloc/memalign/vmalloc/free/vfree trace events can be
used for a few things:
 * Finding memory leaks.
 * Finding malloc/vfree or vmalloc/free mismatches.  The rules are:
malloc/realloc need free, memalign/vmalloc need vfree.  They cannot be
mixed.

Stefan