Re: [RFC 0/3] Introduce a new Qemu machine for RISC-V

[Atish Kumar Patra]

On Thu, May 5, 2022 at 2:37 AM Alistair Francis <alistair23@gmail.com> wrote:
>
> On Tue, May 3, 2022 at 5:57 PM Atish Patra <atishp@atishpatra.org> wrote:
> >
> > On Tue, Apr 19, 2022 at 5:26 PM Atish Patra <atishp@atishpatra.org> wrote:
> > >
> > > On Tue, Apr 19, 2022 at 9:51 AM Daniel P. Berrangé <berrange@redhat.com> 
> > > wrote:
> > > >
> > > > On Mon, Apr 11, 2022 at 07:10:06PM -0700, Atish Patra wrote:
> > > > >
> > > > > The RISC-V virt machine has helped RISC-V software eco system to 
> > > > > evolve at a
> > > > > rapid pace even in absense of the real hardware. It is definitely 
> > > > > commendable.
> > > > > However, the number of devices & commandline options keeps growing as 
> > > > > a result
> > > > > of that as well. That adds flexibility but will also become bit 
> > > > > difficult
> > > > > to manage in the future as more extension support will be added. As 
> > > > > it is the
> > > > > most commonly used qemu machine, it needs to support all kinds of 
> > > > > device and
> > > > > interrupts as well. Moreover, virt machine has limitations on the 
> > > > > maximum
> > > > > number of harts it can support because of all the MMIO devices it has 
> > > > > to support.
> > > > >
> > > > > The RISC-V IMSIC specification allows to develop machines completely 
> > > > > relying
> > > > > on MSI and don't care about the wired interrupts at all. It just 
> > > > > requires
> > > > > all the devices to be present behind a PCI bus or present themselves 
> > > > > as platform
> > > > > MSI device. The former is a more common scenario in x86 world where 
> > > > > most
> > > > > of the devices are behind PCI bus. As there is very limited MMIO 
> > > > > device
> > > > > support, it can also scale to very large number of harts.
> > > > >
> > > > > That's why, this patch series introduces a minimalistic yet very 
> > > > > extensible
> > > > > forward looking machine called as "RISC-V Mini Computer" or "minic". 
> > > > > The
> > > > > idea is to build PC or server like systems with this machine. The 
> > > > > machine can
> > > > > work with or without virtio framework. The current implementation only
> > > > > supports RV64. I am not sure if building a RV32 machine would be of 
> > > > > interest
> > > > > for such machines. The only mmio device it requires is clint to 
> > > > > emulate
> > > > > the mtimecmp.
> > > >
> >
> > Any other thoughts ?
>
> I don't *love* this idea. I think the virt machine is useful, but I'm
> not convinced we need a second one.
>
> This feels a little bit more like a "none" machine, as it contains
> just the bare minimum to work.
>

Ha ha :). That was the whole point. To create a minimal machine that
is easy to maintain and emulate platforms in the real world.

> >
> > > > I would ask what you see as the long term future usage for 'virt' vs
> > > > 'minic' machine types ? Would you expect all existing users of 'virt'
> > > > to ultimately switch to 'minic', or are there distinct non-overlapping
> > > > use cases for 'virt' vs 'minic' such that both end up widely used ?
> > > >
> > >
> > > Nope. I don't expect existing 'virt' users to switch to 'minic' as
> > > they aim to cater to different users.
> > >
> > > Here are the major differences
> > > 1. virt machine supports MMIO devices & wired interrupts. Minic doesn't
>
> This seems like the main difference
>
> > > 2. virt machine doesn't support the MSI only option yet (can be added
> > > though[1]). Minic does.
>
> This could be fixed
>
> > > 3. Number of cpu supported by virt machine are limited because of the
> > > MMIO devices. Minic can scale to very
> > > large numbers of cpu.
>
> Similar to 1
>

Yes. I already had the patch to fix that in the virt machine in the
cover letter.
I did not send it to the mailing list to avoid confusion.

> > > 4. 'Minic' only supports PCI based MSI capable devices. Thus, MSI is a
> > > mandatory requirement for 'minic' while
> > > it is optional for 'virt'.
>
> I'm not fully convinced we need this, but it also doesn't seem to cost
> us a lot in terms of maintenance. It would be beneficial if we could
> share a bit more of the code. Can we share the socket creation code as
> well?
>

Yeah. We can move the socket creation to the common code as well.
There are few others small ones (virt_set/get_aia_guests) can be moved
to common code.

In the future, real world hpc machines may just build their machine
on top of this machine instead of developing from scratch if we allow
some configurability
(e.g memory map, machine name, max cpus etc.)

> I don't like the name minic though. What about something like
> `virt-hpc`, `virt-pcie-minimal` or something like that? That way we
> indicate it's still a virt board
>

Fair enough. I can rename it to virt-hpc or something similar.

> Alistair
>
> > >
> > > 'Minic' aims towards the users who want to create virtual machines
> > > that are MSI based and don't care about
> > > a million options that virt machines provide.  Virt machine is more
> > > complex so that it can be flexible in terms of
> > > what it supports. Minic is a minimalistic machine which doesn't need
> > > to be expanded a lot in the future given that
> > > most of the devices can be behind PCI.
> > >
> > > [1] https://github.com/atishp04/qemu/tree/virt_imsic_only
> > >
> > > > Is 'minic' intended to be able to mimic real physical hardware at all,
> > > > or is it still intended as a purely virtual machine, like a 'virt-ng' ?
> > > >
> > >
> > > Any future hardware that relies only on PCI-MSI based devices, they
> > > can be created on top of minic.
> > > At that point, minic will provide a useful abstract for all those
> > > machines as well. minic doesn't need a virtio framework.
> > > Thus, it can closely emulate such hardware as well.
> > >
> > > > Essentially 'virt' was positioned as the standard machine to use if
> > > > you want to run a virtual machine, without any particular match to
> > > > physical hardware. It feels like 'minic' is creating a second machine
> > > > type to fill the same purpose, so how do users decide which to use ?
> > > >
> > >
> > > I envision 'minic' to be a standard machine for a specific set of user
> > > requirements (x86 style PCI based
> > > machines). Virt machine will continue to be a standard machine for
> > > more generic use cases with MMIO devices.
> > >
> > > > > "Naming is hard". I am not too attached with the name "minic".
> > > > > I just chose least bad one out of the few on my mind :). I am 
> > > > > definitely
> > > > > open to any other name as well.
> > > > >
> > > > > The other alternative to provide MSI only option to aia in the
> > > > > existing virt machine to build MSI only machines. This is certainly 
> > > > > doable
> > > > > and here is the patch that supports that kind of setup.
> > > > >
> > > > > https://github.com/atishp04/qemu/tree/virt_imsic_only
> > > > >
> > > > > However, it even complicates the virt machine even further with 
> > > > > additional
> > > > > command line option, branches in the code. I believe virt machine 
> > > > > will become
> > > > > very complex if we continue this path. I am interested to learn what 
> > > > > everyone
> > > > > else think.
> > > > >
> > > > > It is needless to say that the current version of minic machine
> > > > > is inspired from virt machine and tries to reuse as much as code 
> > > > > possible.
> > > > > The first patch in this series adds MSI support for serial-pci device 
> > > > > so
> > > > > console can work on such a machine. The 2nd patch moves some common 
> > > > > functions
> > > > > between minic and the virt machine to a helper file. The PATCH3 
> > > > > actually
> > > > > implements the new minic machine.
> > > > >
> > > > > I have not added the fw-cfg/flash support. We probably should add 
> > > > > those
> > > > > but I just wanted to start small and get the feedback first.
> > > > > This is a work in progress and have few more TODO items before 
> > > > > becoming the
> > > > > new world order :)
> > > > >
> > > > > 1. OpenSBI doesn't have PCI support. Thus, no console support for 
> > > > > OpenSBI
> > > > > for now.
> > > > > 2. The ns16550 driver in OpenSBI also need to support MSI/MSI-X.
> > > > > 3. Add MSI-X support for serial-pci device.
> > > > >
> > > > > This series can boot Linux distros with the minic machine with or 
> > > > > without virtio
> > > > > devices with out-of-tree Linux kernel patches[1]. Here is an example 
> > > > > commandline
> > > > >
> > > > > Without virtio devices (nvme, serial-pci & e1000e):
> > > > > =====================================================
> > > > > /scratch/workspace/qemu/build/qemu-system-riscv64 -cpu rv64 -M minic 
> > > > > -m 1G -smp 4 -nographic -nodefaults \
> > > > > -display none -bios 
> > > > > /scratch/workspace/opensbi/build/platform/generic/firmware/fw_dynamic.elf
> > > > >  \
> > > > > -kernel /scratch/workspace/linux/arch/riscv/boot/Image \
> > > > > -chardev stdio,mux=on,signal=off,id=charconsole0 \
> > > > > -mon chardev=charconsole0,mode=readline \
> > > > > -device pci-serial,msi=true,chardev=charconsole0 \
> > > > > -drive 
> > > > > id=disk3,file=/scratch/workspace/rootfs_images//fedora/Fedora-Developer-Rawhide-20211110.n.0-sda.raw,format=raw,if=none,id=drive-system-disk,cache=none,format=raw
> > > > >  \
> > > > > -device nvme,serial=deadbeef,drive=disk3 \
> > > > > -netdev user,id=usernet,hostfwd=tcp::10000-:22 -device 
> > > > > e1000e,netdev=usernet,bus=pcie.0 \
> > > > > -append 'root=/dev/nvme0n1p2 rw loglevel=8 memblock=debug 
> > > > > console=ttyS0 earlycon' -d in_asm -D log.txt -s
> > > > >
> > > > > With virtio devices (virtio-scsi-pci, serial-pci & virtio-net-pci)
> > > > > ==================================================================
> > > > > /scratch/workspace/qemu/build/qemu-system-riscv64 -cpu rv64 -M minic 
> > > > > -m 1G -smp 4 -nographic -nodefaults \
> > > > > -display none -bios 
> > > > > /scratch/workspace/opensbi/build/platform/generic/firmware/fw_dynamic.elf
> > > > >  \
> > > > > -kernel /scratch/workspace/linux/arch/riscv/boot/Image \
> > > > > -chardev stdio,mux=on,signal=off,id=charconsole0 \
> > > > > -mon chardev=charconsole0,mode=readline \
> > > > > -device pci-serial,msi=true,chardev=charconsole0 \
> > > > > -drive 
> > > > > file=/scratch/workspace/rootfs_images//fedora/Fedora-Developer-Rawhide-20211110.n.0-sda.raw,format=raw,if=none,id=drive-system-disk,cache=none
> > > > >  \
> > > > > -device virtio-scsi-pci,id=scsi0 -device 
> > > > > scsi-hd,bus=scsi0.0,drive=drive-system-disk,id=system-disk,bootindex=1
> > > > >  \
> > > > > -netdev user,id=n1,hostfwd=tcp::10000-:22 -device 
> > > > > virtio-net-pci,netdev=n1 \
> > > > > -append 'root=/dev/sda2 rw loglevel=8 memblock=debug console=ttyS0 
> > > > > earlycon'
> > > > >
> > > > > The objective of this series is to engage the community to solve this 
> > > > > problem.
> > > > > Please suggest if you have another alternatve solution.
> > > > >
> > > > > [1] https://github.com/atishp04/linux/tree/msi_only_console
> > > > >
> > > > > Atish Patra (3):
> > > > > serial: Enable MSI capablity and option
> > > > > hw/riscv: virt: Move common functions to a separate helper file
> > > > > hw/riscv: Create a new qemu machine for RISC-V
> > > > >
> > > > > configs/devices/riscv64-softmmu/default.mak |   1 +
> > > > > hw/char/serial-pci.c                        |  36 +-
> > > > > hw/riscv/Kconfig                            |  11 +
> > > > > hw/riscv/machine_helper.c                   | 417 +++++++++++++++++++
> > > > > hw/riscv/meson.build                        |   2 +
> > > > > hw/riscv/minic.c                            | 438 ++++++++++++++++++++
> > > > > hw/riscv/virt.c                             | 403 ++----------------
> > > > > include/hw/riscv/machine_helper.h           |  87 ++++
> > > > > include/hw/riscv/minic.h                    |  65 +++
> > > > > include/hw/riscv/virt.h                     |  13 -
> > > > > 10 files changed, 1090 insertions(+), 383 deletions(-)
> > > > > create mode 100644 hw/riscv/machine_helper.c
> > > > > create mode 100644 hw/riscv/minic.c
> > > > > create mode 100644 include/hw/riscv/machine_helper.h
> > > > > create mode 100644 include/hw/riscv/minic.h
> > > > >
> > > > > --
> > > > > 2.25.1
> > > > >
> > > > >
> > > >
> > > > With regards,
> > > > Daniel
> > > > --
> > > > |: https://berrange.com      -o-    
> > > > https://www.flickr.com/photos/dberrange :|
> > > > |: https://libvirt.org         -o-            
> > > > https://fstop138.berrange.com :|
> > > > |: https://entangle-photo.org    -o-    
> > > > https://www.instagram.com/dberrange :|
> > > >
> > > >
> > >
> > >
> > > --
> > > Regards,
> > > Atish
> >
> >
> >
> > --
> > Regards,
> > Atish
> >