Re: [Qemu-ppc] [PATCH v2 1/2] ppc/pnv: Add model for Power8 PHB3 PCIe Host bridge

[Cédric Le Goater]

On 07/27/2018 11:19 AM, Cédric Le Goater wrote:
> On 07/27/2018 10:43 AM, Benjamin Herrenschmidt wrote:
>> On Fri, 2018-07-27 at 10:25 +0200, Cédric Le Goater wrote:
>>> Each PHB creates a pci-bridge device and the PCI bus that comes with it. 
>>> It makes things easier to define PCI devices. 
>>>
>>> It is still quite complex ... Here is a sample :
>>>
>>> qemu-system-ppc64 -m 2G -machine powernv \
>>>   -cpu POWER8 -smp 2,cores=2,threads=1 -accel tcg,thread=multi \
>>>   -kernel ./zImage.epapr -initrd ./rootfs.cpio.xz -bios ./skiboot.lid \
>>>   \
>>>   -device megasas,id=scsi0,bus=pci.0,addr=0x1 \
>>>   -drive 
>>> file=./rhel7-ppc64le.qcow2,if=none,id=drive-scsi0-0-0-0,format=qcow2,cache=none
>>>  \
>>>   -device 
>>> scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=2
>>>  \
>>>   \
>>>   -device ich9-ahci,id=sata0,bus=pci.1,addr=0x1 \
>>>   -drive 
>>> file=./ubuntu-ppc64le.qcow2,if=none,id=drive0,format=qcow2,cache=none \
>>>   -device ide-hd,bus=sata0.0,unit=0,drive=drive0,id=ide,bootindex=1 \
>>>   -device e1000,netdev=net0,mac=C0:FF:EE:00:00:02,bus=pci.1,addr=0x2 \
>>>   -netdev bridge,helper=/usr/libexec/qemu-bridge-helper,br=virbr0,id=net0 \
>>>   -device nec-usb-xhci,bus=pci.1,addr=0x7 \
>>
>> I don't understand why. That means you can't put emulated (or real)
>> PCIe device below it ?
> 
> Well, skiboot does seem to find them. But that's not a good reason.
> I will dig in.

Nothing is wrong in skiboot, that was my misunderstanding of the overall 
device layout.

Each PHB3 device exposes a single PCIE root port on which a *single* device 
can be plugged. It can be a any adapter or a bridge/switch. The QEMU device 
hierarchy is rather deep if you want to have multiple devices on a single 
PHB3 :

dev: pnv-phb3
  bus: phb3-root-bus
    dev: phb3-root-port
      bus: pcie 
         dev: pcie-bridge (only one device allowed) 
            bus: pcie 
               dev: e1000e
               dev: ...


So it makes the command line rather heavy :/ or you need to have 
more phb3s to define an adapter on each. How about 3 per chip ?


Here is a quick survey,

On OpenPower systems: 

  palmetto:  3 PHBs - one 16x, one 8x, one 4x backed by a switch
             with the BMC devices 

  habanero:  3 PHBs - one 16x, one 8x, one 8x backed by a switch 
             with the BMC devices 

  firestone: socket 0 : 2 16x PHBs
             socket 1 : 3 PHBs - one 16x, one 8x, one 8x backed  
                        by a switch with the BMC devices  

  garrison:  socket 0 : 1 16x PHB, 2 8x PHB linked to the GPUs, 1 8x PHB, 
             socket 1 : 1 16x PHB, 2 8x PHB linked to the GPUs, 1 8x PHB
                        backed by a switch with the BMC devices  


On IBM systems :

  tuletas:   2 16x PHB, 2 8x PHB, backed with a switch. 
  

Thanks,

C.