Re: About the performance of hyper-v

[Vitaly Kuznetsov]

Liang Li <liliang324@gmail.com> writes:

>> >> > Analyze events for all VMs, all VCPUs:
>> >> >              VM-EXIT    Samples  Samples%     Time%    Min Time    Max
>> >> > Time         Avg time
>> >> >   EXTERNAL_INTERRUPT     471831    59.89%    68.58%      0.64us
>> >> > 65.42us      2.34us ( +-   0.11% )
>> >> >            MSR_WRITE     238932    30.33%    23.07%      0.48us
>> >> > 41.05us      1.56us ( +-   0.14% )
>> >> >
>> >> > Total Samples:787803, Total events handled time:1611193.84us.
>> >> >
>> >> > I tried turning off hyper-v for the same workload and repeat the test,
>> >> > the overall virtualization overhead reduced by about of 50%:
>> >> >
>> >> > -------
>> >> >
>> >> > Analyze events for all VMs, all VCPUs:
>> >> >
>> >> >              VM-EXIT    Samples  Samples%     Time%    Min Time    Max
>> >> > Time         Avg time
>> >> >           APIC_WRITE     255152    74.43%    50.72%      0.49us
>> >> > 50.01us      1.42us ( +-   0.14% )
>> >> >        EPT_MISCONFIG      39967    11.66%    40.58%      1.55us
>> >> > 686.05us      7.27us ( +-   0.43% )
>> >> >            DR_ACCESS      35003    10.21%     4.64%      0.32us
>> >> > 40.03us      0.95us ( +-   0.32% )
>> >> >   EXTERNAL_INTERRUPT       6622     1.93%     2.08%      0.70us
>> >> > 57.38us      2.25us ( +-   1.42% )
>> >> >
>> >> > Total Samples:342788, Total events handled time:715695.62us.
>> >> >
>> >> > For this scenario,  hyper-v works really bad.  stimer works better
>> >> > than hpet, but on the other hand, it relies on SynIC which has
>> >> > negative effects for IPI intensive workloads.
>> >> > Do you have any plans for improvement?
>> >> >
>> >>
>> >> Hey,
>> >>
>> >> the above can be caused by the fact that when 'hv-synic' is enabled, KVM
>> >> automatically disables APICv and this can explain the overhead and the
>> >> fact that you're seeing more vmexits. KVM disables APICv because SynIC's
>> >> 'AutoEOI' feature is incompatible with it. We can, however, tell Windows
>> >> to not use AutoEOI ('Recommend deprecating AutoEOI' bit) and only
>> >> inhibit APICv if the recommendation was ignored. This is implemented in
>> >> the following KVM patch series:
>> >> https://lore.kernel.org/kvm/20210518144339.1987982-1-vkuznets@redhat.com/
>> >>
>> >> It will, however, require a new 'hv-something' flag to QEMU. For now, it
>> >> can be tested with 'hv-passthrough'.
>> >>
>> >> It would be great if you could give it a spin!
>> >>
>> >> --
>> >> Vitaly
>> >
>> > It's great to know that you already have a solution for this. :)
>> >
>> > By the way,  is there any requirement for the version of windows or
>> > windows updates for the new feature to work?
>>
>> AFAIR, 'Recommend deprecating AutoEOI' bit appeared in WS2012 so I'd
>> expect WS2008 to ignore it completely (and thus SynIC will always be
>> disabling APICv for it).
>>
>
> Hi Vitaly,
>       I tried your patchset and found it's not helpful to reduce the
> virtualization overhead.
> here are some perfdata with the same workload
>
> ===============================
> Analyze events for all VMs, all VCPUs:
>              VM-EXIT    Samples  Samples%     Time%    Min Time    Max
> Time         Avg time
>            MSR_WRITE     924045    89.96%    81.10%      0.42us
> 68.42us      1.26us ( +-   0.07% )
>            DR_ACCESS      44669     4.35%     2.36%      0.32us
> 50.74us      0.76us ( +-   0.32% )
>   EXTERNAL_INTERRUPT      29809     2.90%     6.42%      0.66us
> 70.75us      3.10us ( +-   0.54% )
>               VMCALL      17819     1.73%     5.21%      0.75us
> 15.64us      4.20us ( +-   0.33%
>
> Total Samples:1027227, Total events handled time:1436343.94us.
> ===============================
>
> The result shows the overhead increased.  enable the apicv can help to
> reduce the vm-exit
> caused by interrupt injection, but on the other side, there are a lot
> of vm-exit caused by APIC_EOI.
>
> When turning off the hyper-v and using the kvm apicv, there is no such
> overhead. 

I think I know what's happening. We've asked Windows to use synthetic
MSRs to access APIC (HV_APIC_ACCESS_RECOMMENDED) and this can't be
accelerated in hardware.

Could you please try the following hack (KVM):

diff --git a/arch/x86/kvm/cpuid.c b/arch/x86/kvm/cpuid.c
index c8f2592ccc99..66ee85a83e9a 100644
--- a/arch/x86/kvm/cpuid.c
+++ b/arch/x86/kvm/cpuid.c
@@ -145,6 +145,13 @@ void kvm_update_cpuid_runtime(struct kvm_vcpu *vcpu)
                                           vcpu->arch.ia32_misc_enable_msr &
                                           MSR_IA32_MISC_ENABLE_MWAIT);
        }
+
+       /* Dirty hack: force HV_DEPRECATING_AEOI_RECOMMENDED. Not to be merged! 
*/
+       best = kvm_find_cpuid_entry(vcpu, HYPERV_CPUID_ENLIGHTMENT_INFO, 0);
+       if (best) {
+               best->eax &= ~HV_X64_APIC_ACCESS_RECOMMENDED;
+               best->eax |= HV_DEPRECATING_AEOI_RECOMMENDED;
+       }
 }
 EXPORT_SYMBOL_GPL(kvm_update_cpuid_runtime);
 
> It seems turning on hyper V related features is not always the best
> choice for a windows guest.

Generally it is, we'll just need to make QEMU smarter when setting
'recommendation' bits.

-- 
Vitaly