Re: 5.1.0-rc1 regression: reset fails with kvm and -cpu host

[Jan Kiszka]

On 23.07.20 14:52, Dr. David Alan Gilbert wrote:
> * Vitaly Kuznetsov (vkuznets@redhat.com) wrote:
> > Philippe Mathieu-Daudé <philmd@redhat.com> writes:
> >
> > > +Vitaly
> > >
> > > On 7/23/20 10:40 AM, Dr. David Alan Gilbert wrote:
> > > > * Eduardo Habkost (ehabkost@redhat.com) wrote:
> > > > > On Wed, Jul 22, 2020 at 04:47:32PM -0400, Eduardo Habkost wrote:
> > > > > > On Wed, Jul 22, 2020 at 08:05:01PM +0200, Jan Kiszka wrote:
> > > > > > > On 22.07.20 19:35, Eduardo Habkost wrote:
> > > > > > > > Hi Jan,
> > > > > > > >
> > > > > > > > What was the last version where it worked for you?  Does using
> > > > > > > > "-cpu host,-vmx" help?
> > > > > > >
> > > > > > > Yeah, -vmx does indeed help.
> > > > > > >
> > > > > > > I didn't have the time to bisect yet. Just check my reflog, picked
> > > > > > > eb6490f544, and that works.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > I could reproduce it locally[1], I will bisect it.
> > > > > >
> > > > > > The good news is that "-cpu host,+vmx" still works, on commit
> > > > > > eb6490f544.
> > > > > >
> > > > > > [1] Linux 5.6.19-300.fc32.x86_64, Intel Core i7-8665U CPU.
> > > > >
> > > > > Bisected to:
> > > > >
> > > > > commit b16c0e20c74218f2d69710cedad11da7dd4d2190
> > > > > Author: Paolo Bonzini <pbonzini@redhat.com>
> > > > > Date:   Wed May 20 10:49:22 2020 -0400
> > > > >
> > > > >      KVM: add support for AMD nested live migration
> > > > >
> > > > >      Support for nested guest live migration is part of Linux 5.8, add the
> > > > >      corresponding code to QEMU.  The migration format consists of a few
> > > > >      flags, is an opaque 4k blob.
> > > > >
> > > > >      The blob is in VMCB format (the control area represents the L1 VMCB
> > > > >      control fields, the save area represents the pre-vmentry state; KVM does
> > > > >      not use the host save area since the AMD manual allows that) but QEMU
> > > > >      does not really care about that.  However, the flags need to be
> > > > >      copied to hflags/hflags2 and back.
> > > > >
> > > > >      In addition, support for retrieving and setting the AMD nested 
> > > > > virtualization
> > > > >      states allows the L1 guest to be reset while running a nested guest, but
> > > > >      a small bug in CPU reset needs to be fixed for that to work.
> > > > >
> > > > >      Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> > > >
> > > > Guesswork led me to try reverting the chunk in kvm_put_nested_state;
> > > > without it the reset seems to work; I can't explain that code though.
> >
> > (sorry, missed the beginning of the discussion)
> >
> > So one does:
> >
> > (qemu) system_reset
> >
> > on Intel wiht '-cpu host' and the result is:
> >
> > (qemu) KVM: entry failed, hardware error 0x80000021
>
> Interesting; I hadn't seen that error - I just see a hard hung guest
> rather than a reset one.

I've seen it once or twice, maybe that was also with a more complex 
command line. The point is that an invalid state is very likely loaded 
on reset. Not all invalid states cause KVM to complain, though. Some 
just lock up the guest.

Jan

> (i7-8650U laptop 5.7.9 fedora 32)
>
> Dave
>
> > If you're running a guest on an Intel machine without unrestricted mode
> > support, the failure can be most likely due to the guest entering an invalid
> > state for Intel VT. For example, the guest maybe running in big real mode
> > which is not supported on less recent Intel processors.
> >
> > EAX=00000064 EBX=91df5efe ECX=00000000 EDX=000003f8
> > ESI=00000000 EDI=91ee32c0 EBP=90643260 ESP=00013c68
> > EIP=906428e6 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
> > ES =0000 00000000 0000ffff 00009300
> > CS =f000 ffff0000 0000ffff 00009b00
> > SS =0000 00000000 0000ffff 00009300
> > DS =0000 00000000 0000ffff 00009300
> > FS =0000 00000000 0000ffff 00009300
> > GS =0000 00000000 0000ffff 00009300
> > LDT=0000 00000000 0000ffff 00008200
> > TR =0000 00000000 0000ffff 00008b00
> > GDT=     00000000 0000ffff
> > IDT=     00000000 0000ffff
> > CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
> > DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 
> > DR3=0000000000000000
> > DR6=00000000ffff0ff0 DR7=0000000000000400
> > EFER=0000000000000000
> > Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> ?? ?? ?? 
> > ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ??
> >
> > I can take a look (if no one beats me to it).
> >
> > --
> > Vitaly
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

--
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux