Re: [Qemu-devel] Using the qemu tracepoints with SystemTap

[Stefan Hajnoczi]

On Wed, Sep 14, 2011 at 5:02 PM, William Cohen <address@hidden> wrote:
> On 09/14/2011 10:51 AM, Stefan Hajnoczi wrote:
>> On Tue, Sep 13, 2011 at 8:38 PM, William Cohen <address@hidden> wrote:
>>> On 09/13/2011 12:10 PM, William Cohen wrote:
>>>
>>>> Should the qemu.kvm.cpu_in and qemu.kvm.cpu_out match up?  There are a lot 
>>>> more qemu.kvm.cpu_out than qemu.kvm.cpu_in count.
>>>
>>> I found that cpu_in and cpu_out refer to input and output instructions.  I 
>>> wrote a little script tally up the input and output operations on each port 
>>> to run on a qemu on fc15 machine.
>>>
>>> It generates output like the following:
>>>
>>>
>>> cpu_in
>>>  port    count
>>> 0x01f7     3000
>>> 0x03d5      120
>>> 0xc000     2000
>>> 0xc002     3000
>>>
>>> cpu_out
>>>  port    count
>>> 0x0080      480
>>> 0x01f1     2000
>>> 0x01f2     2000
>>> 0x01f3     2000
>>> 0x01f4     2000
>>> 0x01f5     2000
>>> 0x01f6     2000
>>> 0x01f7     1000
>>> 0x03d4      480
>>> 0x03d5      120
>>> 0x03f6     1000
>>> 0xc000     3000
>>> 0xc002     2000
>>> 0xc004     1000
>>> 0xc090        4
>>>
>>> Looks like lots of touching the ide device ports (0x01f0-0x01ff) and some 
>>> vga controller (0x03d0-0x3df). This is kind of what would be expected when 
>>> the machine is doing a fsck and selinux relabel on the guest virtual 
>>> machines. Look like some pci device access 
>>> (http://www.tech-pro.net/intro_pci.html) also.
>>
>> I think the cpu_in/cpu_out tracing script can be useful in identifying
>> performance problems, especially when obscure guest OSes experience
>> poor performance due to weird ways of interacting with hardware.
>
> Glad this example looks useful.
>
>> Where are you putting your SystemTap scripts?  I suggest creating a
>> public git repo and adding a link from the QEMU wiki tracing page:
>> http://wiki.qemu.org/Features/Tracing
>
> Definitely need a more lasting place to put the QEMU examples. SystemTap has 
> an examples directory which try to put things like this into 
> (http://sourceware.org/systemtap/examples/). These are run as part of the 
> testsuite. These qemu examples are not ready for inclusion in the examples. I 
> have set up https://github.com/wcohen/qemu_systemtap for the qemu examples.

Added a link from http://qemu.org/Features/Tracing.

>> Perhaps we could even include them in a contrib/ or similar directory
>> in qemu.git so that distros can ship them.  But before we worry about
>> that we need a useful set of things that can be observed.
>
> We definitely want to develop scripts that give people a better understanding 
> what is going on. I am just now learning how kvm and qemu work, so the early 
> scripts are just counting events to give people an idea events are frequent. 
> In some cases these could point to issues where some assumptions about event 
> frequency are incorrect. I would like to have scripts that examine the 
> sequences of events and indicate when long latencies occur, but I don't yet 
> have enough understanding of the how kvm and qemu work to implement those.

The vmexit/entry things I mentioned in the mail above are events that
can be traced in the host kernel (kvm:kvm_exit and kvm:kvm_entry).
The QEMU-side equivalent is the ioctl(KVM_RUN) but we do not have any
trace events in kvm-all.c yet (!!).  A trace event before the ioctl
and one after could be used to trace vcpu code execution from QEMU's
perspective.

Stefan