Re: [PATCH 1/2] spapr: number of SMP sockets must be equal to NUMA nodes

[David Gibson]

On Tue, Mar 23, 2021 at 02:21:33PM -0300, Daniel Henrique Barboza wrote:
> 
> 
> On 3/22/21 10:03 PM, David Gibson wrote:
> > On Fri, Mar 19, 2021 at 03:34:52PM -0300, Daniel Henrique Barboza wrote:
> > > Kernel commit 4bce545903fa ("powerpc/topology: Update
> > > topology_core_cpumask") cause a regression in the pseries machine when
> > > defining certain SMP topologies [1]. The reasoning behind the change is
> > > explained in kernel commit 4ca234a9cbd7 ("powerpc/smp: Stop updating
> > > cpu_core_mask"). In short, cpu_core_mask logic was causing troubles with
> > > large VMs with lots of CPUs and was changed by cpu_cpu_mask because, as
> > > far as the kernel understanding of SMP topologies goes, both masks are
> > > equivalent.
> > > 
> > > Further discussions in the kernel mailing list [2] shown that the
> > > powerpc kernel always considered that the number of sockets were equal
> > > to the number of NUMA nodes. The claim is that it doesn't make sense,
> > > for Power hardware at least, 2+ sockets being in the same NUMA node. The
> > > immediate conclusion is that all SMP topologies the pseries machine were
> > > supplying to the kernel, with more than one socket in the same NUMA node
> > > as in [1], happened to be correctly represented in the kernel by
> > > accident during all these years.
> > > 
> > > There's a case to be made for virtual topologies being detached from
> > > hardware constraints, allowing maximum flexibility to users. At the same
> > > time, this freedom can't result in unrealistic hardware representations
> > > being emulated. If the real hardware and the pseries kernel don't
> > > support multiple chips/sockets in the same NUMA node, neither should we.
> > > 
> > > Starting in 6.0.0, all sockets must match an unique NUMA node in the
> > > pseries machine. qtest changes were made to adapt to this new
> > > condition.
> > 
> > Oof.  I really don't like this idea.  It means a bunch of fiddly work
> > for users to match these up, for no real gain.  I'm also concerned
> > that this will require follow on changes in libvirt to not make this a
> > really cryptic and irritating point of failure.
> 
> Haven't though about required Libvirt changes, although I can say that there
> will be some amount to be mande and it will probably annoy existing users
> (everyone that has a multiple socket per NUMA node topology).
> 
> There is not much we can do from the QEMU layer aside from what I've proposed
> here. The other alternative is to keep interacting with the kernel folks to
> see if there is a way to keep our use case untouched.

Right.  Well.. not necessarily untouched, but I'm hoping for more
replies from Cédric to my objections and mpe's.  Even with sockets
being a kinda meaningless concept in PAPR, I don't think tying it to
NUMA nodes makes sense.

> This also means that
> 'ibm,chip-id' will probably remain in use since it's the only place where
> we inform cores per socket information to the kernel.

Well.. unless we can find some other sensible way to convey that
information.  I haven't given up hope for that yet.

-- 
David Gibson                    | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
                                | _way_ _around_!
http://www.ozlabs.org/~dgibson

signature.asc
Description: PGP signature