Re: Libvirt on little.BIG ARM systems unable to start guest if no cpuset is provided

[Michal Prívozník]

On 12/13/21 17:49, Marc Zyngier wrote:
> On Mon, 13 Dec 2021 16:06:14 +0000,
> Peter Maydell <peter.maydell@linaro.org> wrote:
>>
>> KVM on big.little setups is a kernel-level question really; I've
>> cc'd the kvmarm list.
> 
> Thanks Peter for throwing us under the big-little bus! ;-)
> 
>>
>> On Mon, 13 Dec 2021 at 15:02, Qu Wenruo <quwenruo.btrfs@gmx.com> wrote:
>>>
>>>
>>>
>>> On 2021/12/13 21:17, Michal Prívozník wrote:
>>>> On 12/11/21 02:58, Qu Wenruo wrote:
>>>>> Hi,
>>>>>
>>>>> Recently I got my libvirt setup on both RK3399 (RockPro64) and RPI CM4,
>>>>> with upstream kernels.
>>>>>
>>>>> For RPI CM4 its mostly smooth sail, but on RK3399 due to its little.BIG
>>>>> setup (core 0-3 are 4x A55 cores, and core 4-5 are 2x A72 cores), it
>>>>> brings quite some troubles for VMs.
>>>>>
>>>>> In short, without proper cpuset to bind the VM to either all A72 cores
>>>>> or all A55 cores, the VM will mostly fail to boot.
> 
> s/A55/A53/. There were thankfully no A72+A55 ever produced (just the
> though of it makes me sick).
> 
>>>>>
>>>>> Currently the working xml is:
>>>>>
>>>>>    <vcpu placement='static' cpuset='4-5'>2</vcpu>
>>>>>    <cpu mode='host-passthrough' check='none'/>
>>>>>
>>>>> But even with vcpupin, pinning each vcpu to each physical core, VM will
>>>>> mostly fail to start up due to vcpu initialization failed with -EINVAL.
> 
> Disclaimer: I know nothing about libvirt (and no, I don't want to
> know! ;-).
> 
> However, for things to be reliable, you need to taskset the whole QEMU
> process to the CPU type you intend to use. That's because, AFAICT,
> QEMU will snapshot the system registers outside of the vcpu threads,
> and attempt to use the result to configure the actual vcpu threads. If
> they happen to run on different CPU types, the sysregs will differ in
> incompatible ways and an error will be returned. This may or may not
> be a bug, I don't know (I see it as a feature).
> 

Oh, I had no idea that QEMU does this. Anyway, QEMU spawns a thread for
each vCPU which is perfect, because then libvirt can use CGroups to
restrict each vCPU individually onto a subset of physical CPU(s). And in
fact, libvirt has this feature for many years. I always viewed it as a
performance tuning, but TIL.

Michal