Re: Instruction virtual address in TCG Plugins

[Alex Bennée]

Mikhail Tyutin <m.tyutin@yadro.com> writes:

>> >> > I suspect it is because of memory mappings by guest OS that changes 
>> >> > virtual addresses for that block.
>> >> >
>> >> > I also looked at gen_empty_udata_cb() function and considered to extend 
>> >> > plugin API to pass a program counter
>> >> > value as additional callback argument. I thought it would always give 
>> >> > me valid virtual address of an instruction.
>> >> > Unfortunately, I didn't find a way to get value of that register in 
>> >> > architecture agnostic way (it is 'pc' member in
>> >> > CPUArchState structure).
>> >>
>> >> When we merge the register api you should be able to do that. Although
>> >> during testing I realised that PC acted funny compared to everything
>> >> else because we don't actually update the shadow register every
>> >> instruction.
>> >
>> > We implemented similar API to read registers (by coincidence, I posted 
>> > this patch at the same time as the API you
>> > mentioned) and I observe similar behavior. As far as I see, CPU state is 
>> > only updated in between of executed translation
>> > blocks. Switching to 'singlestep' mode helps to fix that, but execution 
>> > overhead is huge.
>> >
>> > There is also blocks 'chaining' mechanism which is likely contributes to 
>> > corrupted blocks vaddr inside of callbacks.
>> > My guess is that 'pc' value for those chained blocks points to the first 
>> > block of entire chain. Unfortunately, It is very
>> > hard to debug, because I can only see block chains when I run whole Linux 
>> > guest OS. Does Qemu has small test
>> > application to trigger long enough chain of translation blocks?
>> 
>> No all registers should be resolved by the end of any block. There is
>> currently no optimisation of register usage between TBs. If you are
>> seeing PC corruption that would be a bug - but fundamentally things
>> would break pretty quick if tb_lookup() and friends didn't have an
>> accurate PC.
>
> I managed to root cause source of corrupted addresses in plugin callbacks.
> There were basically 2 problems:
>
> 1. Memory IO operations force TCG to create special translation blocks to
> process that memory load/store operation. The plugin gets notification for
> this translation block as well, but instrumentation callbacks other than
> memory ones are silently ignored. To make it correct, the plugin has to match
> instruction execution callback from previous TB to memory callback from that
> special TB. The fix was to expose internal ‘memOnly’ TB flag to the plugin to
> handle such TBs differently.

Are you talking about the CF_MEMI_ONLY compile flag? We added this to
avoid double counting executed instructions. Has there been a clash with
the other changes to always cpu_recompile_io? This was a change added to
fix: https://gitlab.com/qemu-project/qemu/-/issues/1866

Richard is going to look at optimising the cpu_recompile_io code so we
"lock in" a shortened translation once we discover a block is doing
MMIO. See https://linaro.atlassian.net/browse/QEMU-605 for an overview.

> 2. Another problem is related to interrupts handling. Since we can insert pre-
> callback on instructions only, the plugin is not aware if instruction is
> actually executed or interrupted by an interrupt or exception. In fact, it
> mistakenly interprets all interrupted instructions as executed. Adding API
> to receive interrupt notification and appropriate handling of it fixes
> the problem.

We don't process any interrupts until the start of each block so no
asynchronous IRQs should interrupt execution. However it is possible
that any given instruction could generate a synchronous exception so if
you need a precise count of execution you need to instrument every
single instruction. With enough knowledge the plugin could avoid
instrumenting stuff that will never fault but that relies on baking
additional knowledge into the plugin.

Generally its only memory operations that can fault (although I guess
FPU and some more esoteric integer ops can).

>
> I will send those patches for review shortly and thank you for dissuading me
> from going to wrong direction!

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro