Re: [PATCH v6 3/3] target/ppc: support single stepping with KVM HV

[Fabiano Rosas]

David Gibson <address@hidden> writes:

> On Tue, Jan 21, 2020 at 05:23:03PM -0300, Fabiano Rosas wrote:
>> David Gibson <address@hidden> writes:
>> 
>> (...)
>> >> > Hrm.... I don't actually see how changing env->msr helps you here.
>> >> > AFAICT if kvm_insert_breakpoint() resolves to kvm_arch_sw_breakpoint()
>> >> > it doesn't rely on the MSR value at all.  If it resolves to
>> >> > kvm_arch_hw_breakpoint(), then it looks like it just stashes
>> >> > information to be pushed into KVM when we re-enter the guest.  None of
>> >> > the information stashed appears to depend on the current MSR, and once
>> >> > we re-enter the MSR will already have been restored.
>> >> >
>> >>
>> >> This is the call chain:
>> >> 
>> >> kvm_arch_insert_sw_breakpoint -> cpu_memory_rw_debug ->
>> >> cpu_get_phys_page_attrs_debug -> ppc_cpu_get_phys_page_debug ->
>> >> ppc64_v3_get_phys_page_debug -> ppc_radix64_get_phys_page_debug:
>> >> 
>> >>     /* Handle Real Mode */
>> >>     if (msr_dr == 0) {
>> >>         /* In real mode top 4 effective addr bits (mostly) ignored */
>> >>         return eaddr & 0x0FFFFFFFFFFFFFFFULL;
>> >>     }
>> >
>> > Ah, right.  Basically the issue is that kvm_insert_breakpoint() takes
>> > an effective address, not a real address, but it might be happening in
>> > a different context than we're executing right now.
>> >
>> > Ok, that makes sense.  Though.. aren't you always inserting the
>> > breakpoint into an interrupt vector?  So wouldn't it always be MMU
>> > off?  Under what circumstances would this get called with mmu_on =
>> > true?
>> 
>> Well, the MSR state at the moment of the breakpoint is that of the
>> currently executing instruction.
>
> Uh... at the moment of setting the breakpoint, or the moment of
> hitting the breakpoint.
>

Setting. I reworded that sentence so many times it actually got worse.

>> So this gets called with mmu_on = true
>> very often because we're often debugging code than runs with
>> IR|DR=1.
>
> Uh... but isn't the whole point here that the state of mmu_on might
> not match the MSR state.  So the two sentences above don't seem to
> mesh together.
>

I meant that they match most of the time but when they don't we hit the
issue we are discussing right now.

> What I think I'm understanding from the code is that in order to *set*
> the breakpoint, you need to set up the MSR to match what you expect it
> will be when you *hit* the breakpoint.  Yes?
>

Yes.

> But since the breakpoint is always placed in an interrupt vector,
> won't that always be real mode?  Or is this one of the vectors that
> can be entered in virtual mode on recent chips?
>

The interrupt sets IR DR according to the AIL value, so it may be
handled in virtual mode as well.

>> However, we could be at a point when IR|DR=1, but the next traced
>> instruction will execute with IR|DR=0. This happens at the rfid at the
>> end of __enter_rtas, for instance.
>> 
>> So ppc_radix64_get_phys_page_debug will check the MSR, see that we are
>> (now) not in real mode and proceed with the page table walk, which could
>> fail.
>> 
>> In the particular case of the __enter_rtas rfid, we have PIDR=1 [1] so
>> if we don't exit ppc_radix64_get_phys_page_debug at the msr_dr == 0
>> check, it will fail to translate the address.
>> 
>> 1 - 
>> https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=eeb715c3e995fbdda0cc05e61216c6c5609bce66
>> 
>> >> Actually, I think there is a bug after ppc_cpu_get_phys_page_debug
>> >> somewhere. There are some cases where GDB wants to read/write to some
>> >> memory, but it gets denied access. Presumably because of one such
>> >> discrepancy as the one above. I need to spend more time looking at this
>> >> to define the problem properly, though.
>> >
>> > Hm, ok.
>> >
>> >> >> +    /*
>> >> >> +     * MSR_SE = 1 will cause a Trace Interrupt in the guest after the
>> >> >> +     * next instruction executes. If this is a rfid, use SRR1 instead
>> >> >> +     * of MSR.
>> >> >> +     */
>> >> >> +    if (rfid) {
>> >> >> +        if ((env->spr[SPR_SRR1] >> MSR_SE) & 1) {
>> >> >> +            /*
>> >> >> +             * The guest is doing a single step itself. Make sure we
>> >> >> +             * restore it later.
>> >> >> +             */
>> >> >> +            env->sstep_kind = SSTEP_GUEST;
>> >> >> +        }
>> >> >> +
>> >> >> +        env->spr[SPR_SRR1] |= (1ULL << MSR_SE);
>> >> >> +        mmu_on = srr1_ir & srr1_dr;
>> >> >
>> >> > s/&/&&/
>> >> >
>> >> 
>> >> Ack.
>> >> 
>> >> >> +    } else {
>> >> >> +        env->msr |= (1ULL << MSR_SE);
>> >> >> +        mmu_on = msr_ir & msr_dr;
>> >> >
>> >> > s/&/&&/
>> >> >
>> >> 
>> >> Ack.
>> >> 
>> >> > Also, what happens if the guest is using MSR[DR] != MSR[IR]?  It's
>> >> > rare, but it is occasionally useful.
>> >> 
>> >> I understand from the ISA that for the purposes of AIL, both bits need
>> >> to be set. So mmu_on = 0 is correct here.
>> >
>> > I'm not sure what you mean by "for the purposes of AIL".
>> >
>> 
>> The reason I'm tracking the translation state here is to be able to tell
>> what will be the value of AIL, since an Alternate Interrupt Location is
>> not used when translation is disabled. In the ISA, under "Alternate
>> Interrupt Location" the only mention of MSR_IR != MSR_DR is:
>> 
>> "Other interrupts that occur when MSR IR=0 or MSR DR=0, are taken as
>> if LPCR AIL=0."
>> 
>> and my interpretation of that text is that AIL value is 0 when IR DR are
>> either 0b00, 0b01 or 0b10.
>> 
>> So "for the purposes of AIL", I'm considering either IR=0 or DR=0 as
>> meaning MMU off.
>
> But AFAICT the mmu_on flag you're setting here has influences other
> than tracking AIL.

The purpose of the mmu_on flag is to know what the AIL value will be at
the time the interrupt happens and therefore know in which mode the
interrupt will be handled, i.e where to put the breakpoint. Remember:

MMU off or AIL = 0 => 0xd00
AIL = 3 => 0xc000000000004d00

With this recent change, I also use it to know what I should temporarily
put in MSR_DR|IR so that the address translation doesn't
fail. Ultimately, the mmu_on flag only influences the placement of the
breakpoint at the Trace Interrupt handler.