Re: Why invtsc (CPUID_APM_INVTSC) is unmigratable?

[Marcelo Tosatti]

On Fri, Jan 26, 2024 at 04:16:57PM +0800, Xiaoyao Li wrote:
> On 1/25/2024 6:05 AM, Marcelo Tosatti wrote:
> > On Wed, Jan 24, 2024 at 10:52:46PM +0800, Xiaoyao Li wrote:
> > > On 1/23/2024 11:39 PM, Marcelo Tosatti wrote:
> > > > On Sat, Jan 20, 2024 at 05:44:07PM +0800, Xiaoyao Li wrote:
> > > > > On 1/20/2024 12:14 AM, Marcelo Tosatti wrote:
> > > > > > On Fri, Jan 19, 2024 at 02:46:22PM +0800, Xiaoyao Li wrote:
> > > > > > > I'm wondering why CPUID_APM_INVTSC is set as unmigratable_flags. 
> > > > > > > Could
> > > > > > > anyone explain it?
> > > > > > 
> > > > > > 
> > > > > > commit 68bfd0ad4a1dcc4c328d5db85dc746b49c1ec07e
> > > > > > Author: Marcelo Tosatti <mtosatti@redhat.com>
> > > > > > Date:   Wed May 14 16:30:09 2014 -0300
> > > > > > 
> > > > > >        target-i386: block migration and savevm if invariant tsc is 
> > > > > > exposed
> > > > > >        Invariant TSC documentation mentions that "invariant TSC 
> > > > > > will run at a
> > > > > >        constant rate in all ACPI P-, C-. and T-states".
> > > > > >        This is not the case if migration to a host with different 
> > > > > > TSC frequency
> > > > > >        is allowed, or if savevm is performed. So block 
> > > > > > migration/savevm.
> > > > > > 
> > > > > > So the rationale here was that without ensuring the destination host
> > > > > > has the same TSC clock frequency, we can't migrate.
> > > > > 
> > > > > It seems to me the concept of invtsc was extended to "tsc freq will 
> > > > > not
> > > > > change even after the machine is live migrated". I'm not sure it is 
> > > > > correct
> > > > > to extend the concept of invtsc.
> > > > > 
> > > > > The main reason of introducing invtsc is to tell the tsc hardware 
> > > > > keeps
> > > > > counting (at the same rate) even at deep C state, so long as other 
> > > > > states.
> > > > > 
> > > > > For example, a guest is created on machine A with X GHz tsc, and 
> > > > > invtsc
> > > > > exposed (machine A can ensure the guest's tsc counts at X GHz at any 
> > > > > state).
> > > > > If the guest is migrated to machine B with Y GHz tsc, and machine B 
> > > > > can also
> > > > > ensure the invtsc of its guest, i.e., the guest's tsc counts at Y GHz 
> > > > > at any
> > > > > state. IMHO, in this case, the invtsc is supported at both src and 
> > > > > dest,
> > > > > which means it is a migratable feature. However, the migration itself 
> > > > > fails,
> > > > > due to mismatched/different configuration of tsc freq, not due to 
> > > > > invtsc.
> > > > > 
> > > > > > However, this was later extended to allow invtsc migratioon when 
> > > > > > setting
> > > > > > tsc-khz explicitly:
> > > > > > 
> > > > > > commit d99569d9d8562c480e0befab601756b0b7b5d0e0
> > > > > > Author: Eduardo Habkost <ehabkost@redhat.com>
> > > > > > Date:   Sun Jan 8 15:32:34 2017 -0200
> > > > > > 
> > > > > >        kvm: Allow invtsc migration if tsc-khz is set explicitly
> > > > > >        We can safely allow a VM to be migrated with invtsc enabled 
> > > > > > if
> > > > > >        tsc-khz is set explicitly, because:
> > > > > >        * QEMU already refuses to start if it can't set the TSC 
> > > > > > frequency
> > > > > >          to the configured value.
> > > > > >        * Management software is already required to keep device
> > > > > >          configuration (including CPU configuration) the same on
> > > > > >          migration source and destination.
> > > > > >        Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> > > > > >        Message-Id: <20170108173234.25721-3-ehabkost@redhat.com>
> > > > > >        Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> > > > > 
> > > > > But in the case that user doesn't set tsc freq explicitly, the live
> > > > > migration is likely to fail or have issues even without invtsc 
> > > > > exposed to
> > > > > guest,
> > > > 
> > > > Depends on how the guest is using the TSC, but yes.
> > > > 
> > > > > if the destination host has a different tsc frequency than src host.
> > > > > 
> > > > > So why bother checking invtsc only?
> > > > 
> > > > Well, if invtsc is exposed to the guest, then it might use the TSC for
> > > > timekeeping purposes.
> > > > 
> > > > Therefore you don't want to fail (on the invtsc clock characteristics)
> > > > otherwise timekeeping in the guest might be problematic.
> > > > 
> > > > But this are all just heuristics.
> > > > 
> > > > Do you have a suggestion for different behaviour?
> > > 
> > > I think we need to block live migration when user doesn't specify a 
> > > certain
> > > tsc frequency explicitly, regardless of invtsc.
> > 
> > Problem is if that guest is using kvmclock for timekeeping, then live 
> > migration
> > is safe (kvmclock logic will update the tsc frequency of the destination
> > host upon migration).
> > 
> 
> It surprises me kvmclock can do it. Cloud you please elaborate it a bit how
> kvmclock achieve it during migration or point me to some codes in Linux?

MSR_KVM_SYSTEM_TIME_NEW description at
Documentation/virt/kvm/x86/msr.rst and

arch/x86/kernel/pvclock.c, __pvclock_clocksource_read function.