Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Daniel P . Berrangé]

On Tue, Dec 14, 2021 at 09:34:18AM +0000, Marc Zyngier wrote:
> On Tue, 14 Dec 2021 00:41:01 +0000,
> Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > 
> > 
> > 
> > On 2021/12/14 00:49, Marc Zyngier wrote:
> > > On Mon, 13 Dec 2021 16:06:14 +0000,
> > > Peter Maydell <peter.maydell@linaro.org> wrote:
> > >> 
> > >> KVM on big.little setups is a kernel-level question really; I've
> > >> cc'd the kvmarm list.
> > > 
> > > Thanks Peter for throwing us under the big-little bus! ;-)
> > > 
> > >> 
> > >> On Mon, 13 Dec 2021 at 15:02, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
> > >>> 
> > >>> 
> > >>> 
> > >>> On 2021/12/13 21:17, Michal Prívozník wrote:
> > >>>> On 12/11/21 02:58, Qu Wenruo wrote:
> > >>>>> Hi,
> > >>>>> 
> > >>>>> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI 
> > >>>>> CM4,
> > >>>>> with upstream kernels.
> > >>>>> 
> > >>>>> For RPI CM4 its mostly smooth sail, but on RK3399 due to its 
> > >>>>> little.BIG
> > >>>>> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
> > >>>>> brings quite some troubles for VMs.
> > >>>>> 
> > >>>>> In short, without proper cpuset to bind the VM to either all A72 cores
> > >>>>> or all A55 cores, the VM will mostly fail to boot.
> > > 
> > > s/A55/A53/. There were thankfully no A72+A55 ever produced (just the
> > > though of it makes me sick).
> > > 
> > >>>>> 
> > >>>>> Currently the working xml is:
> > >>>>> 
> > >>>>>     <vcpu placement='static' cpuset='4-5'>2</vcpu>
> > >>>>>     <cpu mode='host-passthrough' check='none'/>
> > >>>>> 
> > >>>>> But even with vcpupin, pinning each vcpu to each physical core, VM 
> > >>>>> will
> > >>>>> mostly fail to start up due to vcpu initialization failed with 
> > >>>>> -EINVAL.
> > > 
> > > Disclaimer: I know nothing about libvirt (and no, I don't want to
> > > know! ;-).
> > > 
> > > However, for things to be reliable, you need to taskset the whole QEMU
> > > process to the CPU type you intend to use.
> > 
> > Yep, that's what I'm doing.
> 
> Are you sure? The xml directive above seem to only apply to the vcpus,
> and no other QEMU thread.

For historical reasons this XML element is a bit misleadingly named.

With the config

   <vcpu placement='static' cpuset='4-5'>2</vcpu>

the 'cpuset' applies to the QEMU process as a whole - its vCPUs,
I/O threads and any other emulator threads.

There is a separate config for setting per-VCPU binding which was
illustrated elsewhere in this thread.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|