[no subject]

[Alex Bennée]

Subject: Re: riscv64 system emulation maximum core count with virt machine
Date: Fri, 12 Nov 2021 10:16:34 +0000
References: 
<CADJL7eegP=UajgcOxcb=cs1QFim+W3wt0ieq_On6siznEU-byw@mail.gmail.com> 
<D5150EE4-3E9E-43F3-828D-630CA3673A2F@getmailspring.com> 
<CADJL7ecO089N_1GC6GPY5SJmRbVh81i7hE=8nsPZaEDuXHbR2w@mail.gmail.com>
User-agent: mu4e 1.7.4; emacs 28.0.60
In-reply-to: 
<CADJL7ecO089N_1GC6GPY5SJmRbVh81i7hE=8nsPZaEDuXHbR2w@mail.gmail.com>


Mark Wyse <wysem@cs.washington.edu> writes:

> Thanks Rahul,
>
> After some digging I came to the same conclusions. I'll dig deeper to see if 
> there are any impacts in raising max CPU count for RISC-V Virt
> machine, but I definitely agree that it should be set higher.

If you look at the commit that added the #define:

18df0b4695 (hw/riscv: virt: Allow creating multiple NUMA sockets)

You'll see amongst other things the number of vCPUs affected the size of
the one of the regions (IRQ controller?):

     [VIRT_RTC] =         {   0x101000,        0x1000 },
     [VIRT_CLINT] =       {  0x2000000,       0x10000 },
     [VIRT_PCIE_PIO] =    {  0x3000000,       0x10000 },
-    [VIRT_PLIC] =        {  0xc000000,     0x4000000 },
+    [VIRT_PLIC] =        {  0xc000000, VIRT_PLIC_SIZE(VIRT_CPUS_MAX * 2) },
     [VIRT_UART0] =       { 0x10000000,         0x100 },
     [VIRT_VIRTIO] =      { 0x10001000,        0x1000 },

which I suspects limits the total number you can handle without messing
around with the MMIO memory map.

> FWIW, I was able to run a 32-core kernel after adjusting code and kernel 
> appropriately and don't recall any warning or error messages
> popping up.

I should also point out from a emulation performance point of view you
will run into diminishing returns as you increases the number of CPUs as
the emulator has to spend more time ensuring cross-vCPU consistency when
things like the TLBs are flushed.

>
> Best,
> Mark
>
> On Wed, Sep 29, 2021 at 10:16 AM Rahul Pathak <rpathakmailbox@gmail.com> 
> wrote:
>
>  If you want to boot with more than 8 harts then you also need to increase 
> the CONFIG_NR_CPUS in the kernel. Along with change in
>  QEMU - VIRT_CPUS_MAX in  "include/hw/riscv/virt.h" 
>
>  But, the limit of 8 is low and should be increased. Don't know the reason 
> why its 8.
>
>  On Sep 22 2021, at 10:19 pm, Mark Wyse <wysem@cs.washington.edu> wrote:
>
>  Hello all,
>
>  Apologies if I missed the answer in the docs, but I am wondering why the 
> virt machine is limited to only 8 cores when running
>  RISC-V 64 system emulation? I am interested in booting a 16, 24, or even 32 
> core SMP Linux kernel with OpenSBI as the
>  bootloader on the generic virt machine before adding a custom machine 
> definition. I was able to successfully boot Linux kernel
>  5.9.0 with OpenSBI v0.9 and 8 cores/harts with an ext4 filesystem provided 
> by Buildroot.
>
>  I poked around the docs and source code a bit, and found mention of the core 
> count limit in the docs
>  (https://qemu.readthedocs.io/en/latest/system/riscv/virt.html), but no 
> reason as to why this is the limit. It also appears that
>  providing a custom device tree blob and setting QEMU's -smp option to match 
> might overcome this limitation, but it would be
>  better if more than 8 cores was supported directly.
>
>  Best,
>  Mark
>
>  Mark Wyse
>  pronouns: he/him/his
>  PhD Student
>  Paul G. Allen School of Computer Science & Engineering
>  University of Washington


-- 
Alex Bennée