[Bug 1920013] Re: Unable to pass-through PCIe devices from a ppc64le host to an x86_64 guest

[cyrozap]

I've done some more investigating, and have produced a backtrace of the
error:

    #0  0x00003ffff6b63228 in __libc_signal_restore_set (set=0x3fffffffcec8) at 
../sysdeps/unix/sysv/linux/internal-signals.h:84
    #1  0x00003ffff6b63228 in __GI_raise (sig=<optimized out>) at 
../sysdeps/unix/sysv/linux/raise.c:48
    #2  0x00003ffff6b4358c in __GI_abort () at abort.c:79
    #3  0x000000010080d524 in object_dynamic_cast_assert
        (obj=0x1016db860, typename=0x100bf9980 "qemu:iommu-memory-region", 
file=0x100bf9940 "/usr/src/qemu/include/exec/memory.h", line=<optimized out>, 
func=0x100c08a70 <__func__.21845> "IOMMU_MEMORY_REGION") at ../qom/object.c:883
    #4  0x00000001006b6f84 in IOMMU_MEMORY_REGION (obj=<optimized out>) at 
/usr/src/qemu/include/exec/memory.h:42
    #5  0x00000001006b6f84 in vfio_spapr_create_window (container=0x102357380, 
section=0x3fffffffd410, pgsize=0x3fffffffd368)
        at ../hw/vfio/spapr.c:149
    #6  0x00000001007a09a0 in vfio_listener_region_add (listener=0x102357390, 
section=0x3fffffffd410) at ../hw/vfio/common.c:709
    #7  0x00000001006ea6f4 in listener_add_address_space (as=<optimized out>, 
listener=0x102357390) at ../softmmu/memory.c:2729
    #8  0x00000001006ea6f4 in memory_listener_register (listener=0x102357390, 
as=<optimized out>) at ../softmmu/memory.c:2796
    #9  0x00000001007a36f4 in vfio_connect_container (errp=0x3fffffffe818, 
as=<optimized out>, group=0x102357300)
        at ../hw/vfio/common.c:1886
    #10 0x00000001007a36f4 in vfio_get_group (groupid=<optimized out>, 
as=<optimized out>, errp=0x3fffffffe818)
        at ../hw/vfio/common.c:2003
    #11 0x000000010071a2a8 in vfio_realize (pdev=0x102350f80, 
errp=0x3fffffffe818) at ../hw/vfio/pci.c:2834
    #12 0x0000000100488e20 in pci_qdev_realize (qdev=0x102350f80, 
errp=0x3fffffffe940) at ../hw/pci/pci.c:2113
    #13 0x00000001008063e0 in device_set_realized (obj=0x102350f80, 
value=<optimized out>, errp=0x3fffffffea50) at ../hw/core/qdev.c:761
    #14 0x000000010080afbc in property_set_bool
        (obj=0x102350f80, v=<optimized out>, name=<optimized out>, 
opaque=0x1014f1930, errp=0x3fffffffea50) at ../qom/object.c:2257
    #15 0x000000010080ee2c in object_property_set (obj=0x102350f80, 
name=0x100c023a0 "realized", v=
        0x102351d30, errp=0x101450b30 <error_fatal>) at ../qom/object.c:1402
    #16 0x000000010080a55c in object_property_set_qobject
        (obj=0x102350f80, name=0x100c023a0 "realized", value=<optimized out>, 
errp=0x101450b30 <error_fatal>) at ../qom/qom-qobject.c:28
    #17 0x000000010080f1b0 in object_property_set_bool
        (obj=0x102350f80, name=0x100c023a0 "realized", value=<optimized out>, 
errp=0x101450b30 <error_fatal>) at ../qom/object.c:1472
    #18 0x00000001008042bc in qdev_realize (dev=0x102350f80, bus=<optimized 
out>, errp=0x101450b30 <error_fatal>)
        at ../hw/core/qdev.c:389
    #19 0x000000010036cfac in qdev_device_add (opts=0x1014e9960, 
errp=0x101450b30 <error_fatal>)
        at /usr/src/qemu/include/hw/qdev-core.h:17
    #20 0x00000001006d5e68 in device_init_func (opaque=<optimized out>, 
opts=<optimized out>, errp=<optimized out>)
        at ../softmmu/vl.c:1202
    #21 0x0000000100abe070 in qemu_opts_foreach
        (list=<optimized out>, func=0x1006d5e40 <device_init_func>, opaque=0x0, 
errp=0x101450b30 <error_fatal>)
        at ../util/qemu-option.c:1167
    #22 0x00000001006da110 in qemu_create_cli_devices () at ../softmmu/vl.c:2494
    #23 0x00000001006da110 in qmp_x_exit_preconfig (errp=<optimized out>) at 
../softmmu/vl.c:2542
    #24 0x00000001006df87c in qemu_init (argc=<optimized out>, argv=<optimized 
out>, envp=<optimized out>) at ../softmmu/vl.c:3553
    #25 0x000000010031d3c8 in main (argc=<optimized out>, argv=<optimized out>, 
envp=<optimized out>) at ../softmmu/main.c:49

I also took a look at some of the arguments in frame #5 (where the type
check is failing):

    (gdb) f
    #5  vfio_spapr_create_window (container=0x102357380, 
section=0x3fffffffd410, pgsize=0x3fffffffd368) at ../hw/vfio/spapr.c:149
    149         IOMMUMemoryRegion *iommu_mr = IOMMU_MEMORY_REGION(section->mr);
    (gdb) l
    144     int vfio_spapr_create_window(VFIOContainer *container,
    145                                  MemoryRegionSection *section,
    146                                  hwaddr *pgsize)
    147     {
    148         int ret = 0;
    149         IOMMUMemoryRegion *iommu_mr = IOMMU_MEMORY_REGION(section->mr);
    150         uint64_t pagesize = 
memory_region_iommu_get_min_page_size(iommu_mr), pgmask;
    151         unsigned entries, bits_total, bits_per_level, max_levels;
    152         struct vfio_iommu_spapr_tce_create create = { .argsz = 
sizeof(create) };
    153         long rampagesize = qemu_minrampagesize();
    (gdb) p *section->mr
    $5 = {parent_obj = {class = 0x101512f90, free = 0x0, Python Exception 
<class 'gdb.error'> There is no member named keys.: 
    properties = 0x101716b60, ref = 1, parent = 0x1016db800}, romd_mode = true, 
      ram = true, subpage = false, readonly = false, nonvolatile = false, 
rom_device = false, flush_coalesced_mmio = false, 
      dirty_log_mask = 0 '\000', is_iommu = false, ram_block = 0x101725c30, 
owner = 0x1016db800, ops = 0x101300b00 <unassigned_mem_ops>, 
      opaque = 0x0, container = 0x0, size = 134217728, addr = 0, destructor = 
0x1006e0d50 <memory_region_destructor_ram>, 
      align = 2097152, terminates = true, ram_device = false, enabled = true, 
warning_printed = false, vga_logging_count = 0 '\000', 
      alias = 0x0, alias_offset = 0, priority = 0, subregions = {tqh_first = 
0x0, tqh_circ = {tql_next = 0x0, tql_prev = 0x1016db908}}, 
      subregions_link = {tqe_next = 0x0, tqe_circ = {tql_next = 0x0, tql_prev = 
0x0}}, coalesced = {tqh_first = 0x0, tqh_circ = {
          tql_next = 0x0, tql_prev = 0x1016db928}}, name = 0x101725b10 
"pc.ram", ioeventfd_nb = 0, ioeventfds = 0x0}

So it seems this is failing because the memory region "pc.ram" is not an
IOMMU ("is_iommu = false"). I'm not really sure what that means, and I
still don't know how to fix this, but hopefully this information will
help.

If there's any more information I should provide, please let me know.

-- 
You received this bug notification because you are a member of qemu-
devel-ml, which is subscribed to QEMU.
https://bugs.launchpad.net/bugs/1920013

Title:
  Unable to pass-through PCIe devices from a ppc64le host to an x86_64
  guest

Status in QEMU:
  New

Bug description:
  Attempting to pass through a PCIe device from a ppc64le host to an
  x86_64 guest with QEMU v5.2.0-3031-g571d413b5d (built from git master)
  fails with the following error:

      include/exec/memory.h:43:IOMMU_MEMORY_REGION: Object 0x10438eb00
  is not an instance of type qemu:iommu-memory-region

  To reproduce this issue, simply run the following command on a POWER9
  system:

      qemu-system-x86_64 -machine q35 -device vfio-pci,host=$DBSF

  Where $DBSF is a domain:bus:slot.function PCIe device address.

  This also fails with QEMU 3.1.0 (from Debian Buster), so I assume this
  has never worked. Helpfully, the error message it prints seems to
  indicate where the problem is:

      hw/vfio/spapr.c:147:vfio_spapr_create_window: Object 0x164473510
  is not an instance of type qemu:iommu-memory-region

  My kernel (Linux v5.8.0 plus some small unrelated patches) is built
  with the page size set to 4k, so this issue shouldn't be due to a page
  size mismatch. And as I stated earlier, my host arch is ppc64le, so it
  shouldn't be an endianness issue, either.

  I assume this should be possible (in theory) since I've seen reports
  of others getting PCIe passthrough working with aarch64 guests on
  x86_64 hosts, but of course that (passthrough to weird guest arch on
  x86) is somewhat the opposite of what I'm trying to do (passthrough to
  x86 on weird host arch) so I don't know for sure. If it is possible,
  I'm willing to develop a fix myself, but I'm almost completely
  unfamiliar with QEMU's internals so if anyone has any advice on where
  to start I'd greatly appreciate it.

To manage notifications about this bug go to:
https://bugs.launchpad.net/qemu/+bug/1920013/+subscriptions