Re: About the kvm-no-adjvtime CPU property

[Ying Fang]

On 6/3/2020 4:53 PM, Andrew Jones wrote:
> On Tue, Jun 02, 2020 at 03:47:22PM +0800, Ying Fang wrote:
> > On 2020/6/1 20:29, Andrew Jones wrote:
> > > On Mon, Jun 01, 2020 at 08:07:31PM +0800, Ying Fang wrote:
> > > > On 2020/6/1 16:07, Andrew Jones wrote:
> > > > > On Sat, May 30, 2020 at 04:56:26PM +0800, Ying Fang wrote:
> > > > > > About the kvm-no-adjvtime CPU property
> > > > > >
> > > > > > Hi Andrew,
> > > > > > To adjust virutal time, a new kvm cpu property kvm-no-adjvtime
> > > > > > was introduced to 5.0 virt machine types. However the cpu
> > > > > > property was enabled only for host-passthrough and max cpu model.
> > > > > > As for other cpu model like cortex-a57, cortex-a53, cortex-a72,
> > > > > > this kvm-adjvtime is not enabled by default, which means the
> > > > > > virutal time can not be adjust for them.
> > > > > >
> > > > > > Here, for example, if VM is configured with kvm enabled:
> > > > > >
> > > > > >      <cpu mode='custom' match='exact' check='partial'>
> > > > > >        <model fallback='allow'>cortex-a72</model>
> > > > > >        <topology sockets='2' dies='1' cores='2' threads='1'/>
> > > > > >        <numa>
> > > > > >          <cell id='0' cpus='0-1' memory='16777216' unit='KiB'/>
> > > > > >          <cell id='1' cpus='2-3' memory='16777216' unit='KiB'/>
> > > > > >        </numa>
> > > > > >      </cpu>
> > > > > >
> > > > > > We cannot adjust virtual time even if 5.0 virt machine is used.
> > > > > > So i'd like to add it to other cpu model, do you have any
> > > > > > suggestions here ?
> > > > >
> > > > > Hi Fang,
> > > > >
> > > > > The cpu feature only requires kvm.  If a cpu model may be used with kvm,
> > > > > then the feature can be allowed to be used with the model.  What I find
> > > > > interesting is that the cpu model is being used with kvm instead of 'host'
> > > > > or 'max'.  Can you explain the reasons for that?  Currently, when using
> > > > yes，the cpu model is indeed used with kvm.
> > > >
> > > > There is a situation where the host cpu model is Cortex-A72 and
> > > > a 'custom' cpu mode is used to keep insrtuction set compatible between
> > > > the source and destination host machine when doing live migration.
> > > > So the host physical machine cpu model is Cortex-A72 but
> > > > host-passthrough model is mode used here.
> > I mean host-passthrough model is 'not' used here. Sorry to make it
> > confusing.
>
> I guessed as much.
>
> > > Are the source and destinations hosts used in the migration identical?
> > > If so, then the guest can use cpu 'host' and disable cpu features that
> > > should not be exposed (e.g. -cpu host,pmu=off).  If the source and
> > > destination hosts are not identical, then I'm curious what those exact
> > > differences are.  With the way AArch64 KVM works today, even using the
> > > Cortex-A72 cpu model should require identical hosts when migrating.  Or,
> > > at least both hosts must be compatible with Cortex-A72 and any difference
> > > in ID registers must be somehow getting hidden from the guest.
> > Yes, you are right.
> > We have AAarch64 server with cpu based on Cortex-A72 and some extra
> > instruction set added. Source host with cpu based on V1 and destination host
> > with cpu based on V2 and they are compatible with Cortex-A72. We want to use
> > a 'custom' cpu mode here to make it possible to do live migration between
> > them. This is the scenario where the 'host' cpu model is not used since a
> > 'custom' cpu model Cortex-A72 is used here .
>
> What you've described here is indeed the reason to use CPU models. I.e.
> enabling the migration from a host of one type to another by creating a
> guest that only enables the features contained in both hosts (as well as
> maintaining all state that describes the CPU type, e.g. MIDR).
> Unfortunately, unless your KVM has patches that aren't upstream, then that
> doesn't work on AArch64 KVM (more on that below). It may appear to be
> working for you, because your guest kernel and userspace don't mind the
> slight differences exposed to it between the hosts, or those differences
> are limited to explicitly disabled features. If that's the case, then I
> would guess that using '-cpu host' and disabling the same features would
> "work" as well.

Yes, upstream KVM currently does not support it. We are planning to add
support for the aarch64 platform since we have the situation where it is
needed for our hardware.

@Marc Zyngier, is there anyone who doing on this?
> Here's some more details on why the Cortex-A72 CPU model doesn't matter
> with upstream KVM. First, upstream AArch64 KVM doesn't support CPU models,
> and it doesn't even have a Cortex-A72 preferred target. For Cortex-A72
> it will use "KVM_ARM_TARGET_GENERIC_V8", which is the same thing 'host'
> would do when running on a Cortex-A72. Second, if V2 of the Cortex-A72-
> based CPU you're using changed the revision of the MIDR, or any other
> state that gets passed directly through to the guest like the MIDR, then
> that state will change on migration. If a guest looks before migration and
> again after migration, then it could get confused. A guest kernel may only
> look once on boot and then not notice, but anything exposed to userspace
> is extra risky, as userspace may check more frequently.

Yes, just as explained here.

> In short, without KVM patches that aren't upstream, then it's risky to
> migrate between machines with V1 and V2 of these CPUs. And, it doesn't
> help to use the Cortex-A72 CPU model.
Thanks for your detailed introduction.

> Thanks,
> drew
>
>
> > However the kvm-adjvtime
> > feature is also need. So I think we should move kvm_arm_add_vcpu_properties
> > to arm_cpu_post_init, instead of limited to 'host' and 'max' cpu model[1].
> >
> > 1: https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg00091.html
> > > Thanks,
> > > drew
>
>
> .