Re: [RFC 0/3] Introduce a new Qemu machine for RISC-V

[Alistair Francis]

On Tue, May 3, 2022 at 5:57 PM Atish Patra <atishp@atishpatra.org> wrote:
>
> On Tue, Apr 19, 2022 at 5:26 PM Atish Patra <atishp@atishpatra.org> wrote:
> >
> > On Tue, Apr 19, 2022 at 9:51 AM Daniel P. Berrangé <berrange@redhat.com> 
> > wrote:
> > >
> > > On Mon, Apr 11, 2022 at 07:10:06PM -0700, Atish Patra wrote:
> > > >
> > > > The RISC-V virt machine has helped RISC-V software eco system to evolve 
> > > > at a
> > > > rapid pace even in absense of the real hardware. It is definitely 
> > > > commendable.
> > > > However, the number of devices & commandline options keeps growing as a 
> > > > result
> > > > of that as well. That adds flexibility but will also become bit 
> > > > difficult
> > > > to manage in the future as more extension support will be added. As it 
> > > > is the
> > > > most commonly used qemu machine, it needs to support all kinds of 
> > > > device and
> > > > interrupts as well. Moreover, virt machine has limitations on the 
> > > > maximum
> > > > number of harts it can support because of all the MMIO devices it has 
> > > > to support.
> > > >
> > > > The RISC-V IMSIC specification allows to develop machines completely 
> > > > relying
> > > > on MSI and don't care about the wired interrupts at all. It just 
> > > > requires
> > > > all the devices to be present behind a PCI bus or present themselves as 
> > > > platform
> > > > MSI device. The former is a more common scenario in x86 world where most
> > > > of the devices are behind PCI bus. As there is very limited MMIO device
> > > > support, it can also scale to very large number of harts.
> > > >
> > > > That's why, this patch series introduces a minimalistic yet very 
> > > > extensible
> > > > forward looking machine called as "RISC-V Mini Computer" or "minic". The
> > > > idea is to build PC or server like systems with this machine. The 
> > > > machine can
> > > > work with or without virtio framework. The current implementation only
> > > > supports RV64. I am not sure if building a RV32 machine would be of 
> > > > interest
> > > > for such machines. The only mmio device it requires is clint to emulate
> > > > the mtimecmp.
> > >
>
> Any other thoughts ?

I don't *love* this idea. I think the virt machine is useful, but I'm
not convinced we need a second one.

This feels a little bit more like a "none" machine, as it contains
just the bare minimum to work.

>
> > > I would ask what you see as the long term future usage for 'virt' vs
> > > 'minic' machine types ? Would you expect all existing users of 'virt'
> > > to ultimately switch to 'minic', or are there distinct non-overlapping
> > > use cases for 'virt' vs 'minic' such that both end up widely used ?
> > >
> >
> > Nope. I don't expect existing 'virt' users to switch to 'minic' as
> > they aim to cater to different users.
> >
> > Here are the major differences
> > 1. virt machine supports MMIO devices & wired interrupts. Minic doesn't

This seems like the main difference

> > 2. virt machine doesn't support the MSI only option yet (can be added
> > though[1]). Minic does.

This could be fixed

> > 3. Number of cpu supported by virt machine are limited because of the
> > MMIO devices. Minic can scale to very
> > large numbers of cpu.

Similar to 1

> > 4. 'Minic' only supports PCI based MSI capable devices. Thus, MSI is a
> > mandatory requirement for 'minic' while
> > it is optional for 'virt'.

I'm not fully convinced we need this, but it also doesn't seem to cost
us a lot in terms of maintenance. It would be beneficial if we could
share a bit more of the code. Can we share the socket creation code as
well?

I don't like the name minic though. What about something like
`virt-hpc`, `virt-pcie-minimal` or something like that? That way we
indicate it's still a virt board

Alistair

> >
> > 'Minic' aims towards the users who want to create virtual machines
> > that are MSI based and don't care about
> > a million options that virt machines provide.  Virt machine is more
> > complex so that it can be flexible in terms of
> > what it supports. Minic is a minimalistic machine which doesn't need
> > to be expanded a lot in the future given that
> > most of the devices can be behind PCI.
> >
> > [1] https://github.com/atishp04/qemu/tree/virt_imsic_only
> >
> > > Is 'minic' intended to be able to mimic real physical hardware at all,
> > > or is it still intended as a purely virtual machine, like a 'virt-ng' ?
> > >
> >
> > Any future hardware that relies only on PCI-MSI based devices, they
> > can be created on top of minic.
> > At that point, minic will provide a useful abstract for all those
> > machines as well. minic doesn't need a virtio framework.
> > Thus, it can closely emulate such hardware as well.
> >
> > > Essentially 'virt' was positioned as the standard machine to use if
> > > you want to run a virtual machine, without any particular match to
> > > physical hardware. It feels like 'minic' is creating a second machine
> > > type to fill the same purpose, so how do users decide which to use ?
> > >
> >
> > I envision 'minic' to be a standard machine for a specific set of user
> > requirements (x86 style PCI based
> > machines). Virt machine will continue to be a standard machine for
> > more generic use cases with MMIO devices.
> >
> > > > "Naming is hard". I am not too attached with the name "minic".
> > > > I just chose least bad one out of the few on my mind :). I am definitely
> > > > open to any other name as well.
> > > >
> > > > The other alternative to provide MSI only option to aia in the
> > > > existing virt machine to build MSI only machines. This is certainly 
> > > > doable
> > > > and here is the patch that supports that kind of setup.
> > > >
> > > > https://github.com/atishp04/qemu/tree/virt_imsic_only
> > > >
> > > > However, it even complicates the virt machine even further with 
> > > > additional
> > > > command line option, branches in the code. I believe virt machine will 
> > > > become
> > > > very complex if we continue this path. I am interested to learn what 
> > > > everyone
> > > > else think.
> > > >
> > > > It is needless to say that the current version of minic machine
> > > > is inspired from virt machine and tries to reuse as much as code 
> > > > possible.
> > > > The first patch in this series adds MSI support for serial-pci device so
> > > > console can work on such a machine. The 2nd patch moves some common 
> > > > functions
> > > > between minic and the virt machine to a helper file. The PATCH3 actually
> > > > implements the new minic machine.
> > > >
> > > > I have not added the fw-cfg/flash support. We probably should add those
> > > > but I just wanted to start small and get the feedback first.
> > > > This is a work in progress and have few more TODO items before becoming 
> > > > the
> > > > new world order :)
> > > >
> > > > 1. OpenSBI doesn't have PCI support. Thus, no console support for 
> > > > OpenSBI
> > > > for now.
> > > > 2. The ns16550 driver in OpenSBI also need to support MSI/MSI-X.
> > > > 3. Add MSI-X support for serial-pci device.
> > > >
> > > > This series can boot Linux distros with the minic machine with or 
> > > > without virtio
> > > > devices with out-of-tree Linux kernel patches[1]. Here is an example 
> > > > commandline
> > > >
> > > > Without virtio devices (nvme, serial-pci & e1000e):
> > > > =====================================================
> > > > /scratch/workspace/qemu/build/qemu-system-riscv64 -cpu rv64 -M minic -m 
> > > > 1G -smp 4 -nographic -nodefaults \
> > > > -display none -bios 
> > > > /scratch/workspace/opensbi/build/platform/generic/firmware/fw_dynamic.elf
> > > >  \
> > > > -kernel /scratch/workspace/linux/arch/riscv/boot/Image \
> > > > -chardev stdio,mux=on,signal=off,id=charconsole0 \
> > > > -mon chardev=charconsole0,mode=readline \
> > > > -device pci-serial,msi=true,chardev=charconsole0 \
> > > > -drive 
> > > > id=disk3,file=/scratch/workspace/rootfs_images//fedora/Fedora-Developer-Rawhide-20211110.n.0-sda.raw,format=raw,if=none,id=drive-system-disk,cache=none,format=raw
> > > >  \
> > > > -device nvme,serial=deadbeef,drive=disk3 \
> > > > -netdev user,id=usernet,hostfwd=tcp::10000-:22 -device 
> > > > e1000e,netdev=usernet,bus=pcie.0 \
> > > > -append 'root=/dev/nvme0n1p2 rw loglevel=8 memblock=debug console=ttyS0 
> > > > earlycon' -d in_asm -D log.txt -s
> > > >
> > > > With virtio devices (virtio-scsi-pci, serial-pci & virtio-net-pci)
> > > > ==================================================================
> > > > /scratch/workspace/qemu/build/qemu-system-riscv64 -cpu rv64 -M minic -m 
> > > > 1G -smp 4 -nographic -nodefaults \
> > > > -display none -bios 
> > > > /scratch/workspace/opensbi/build/platform/generic/firmware/fw_dynamic.elf
> > > >  \
> > > > -kernel /scratch/workspace/linux/arch/riscv/boot/Image \
> > > > -chardev stdio,mux=on,signal=off,id=charconsole0 \
> > > > -mon chardev=charconsole0,mode=readline \
> > > > -device pci-serial,msi=true,chardev=charconsole0 \
> > > > -drive 
> > > > file=/scratch/workspace/rootfs_images//fedora/Fedora-Developer-Rawhide-20211110.n.0-sda.raw,format=raw,if=none,id=drive-system-disk,cache=none
> > > >  \
> > > > -device virtio-scsi-pci,id=scsi0 -device 
> > > > scsi-hd,bus=scsi0.0,drive=drive-system-disk,id=system-disk,bootindex=1 \
> > > > -netdev user,id=n1,hostfwd=tcp::10000-:22 -device 
> > > > virtio-net-pci,netdev=n1 \
> > > > -append 'root=/dev/sda2 rw loglevel=8 memblock=debug console=ttyS0 
> > > > earlycon'
> > > >
> > > > The objective of this series is to engage the community to solve this 
> > > > problem.
> > > > Please suggest if you have another alternatve solution.
> > > >
> > > > [1] https://github.com/atishp04/linux/tree/msi_only_console
> > > >
> > > > Atish Patra (3):
> > > > serial: Enable MSI capablity and option
> > > > hw/riscv: virt: Move common functions to a separate helper file
> > > > hw/riscv: Create a new qemu machine for RISC-V
> > > >
> > > > configs/devices/riscv64-softmmu/default.mak |   1 +
> > > > hw/char/serial-pci.c                        |  36 +-
> > > > hw/riscv/Kconfig                            |  11 +
> > > > hw/riscv/machine_helper.c                   | 417 +++++++++++++++++++
> > > > hw/riscv/meson.build                        |   2 +
> > > > hw/riscv/minic.c                            | 438 ++++++++++++++++++++
> > > > hw/riscv/virt.c                             | 403 ++----------------
> > > > include/hw/riscv/machine_helper.h           |  87 ++++
> > > > include/hw/riscv/minic.h                    |  65 +++
> > > > include/hw/riscv/virt.h                     |  13 -
> > > > 10 files changed, 1090 insertions(+), 383 deletions(-)
> > > > create mode 100644 hw/riscv/machine_helper.c
> > > > create mode 100644 hw/riscv/minic.c
> > > > create mode 100644 include/hw/riscv/machine_helper.h
> > > > create mode 100644 include/hw/riscv/minic.h
> > > >
> > > > --
> > > > 2.25.1
> > > >
> > > >
> > >
> > > With regards,
> > > Daniel
> > > --
> > > |: https://berrange.com      -o-    
> > > https://www.flickr.com/photos/dberrange :|
> > > |: https://libvirt.org         -o-            
> > > https://fstop138.berrange.com :|
> > > |: https://entangle-photo.org    -o-    
> > > https://www.instagram.com/dberrange :|
> > >
> > >
> >
> >
> > --
> > Regards,
> > Atish
>
>
>
> --
> Regards,
> Atish
>