Re: [Qemu-devel] [PATCH v5 11/36] spapr/xive: use the VCPU id as a NVT identifier

[Cédric Le Goater]

>>>>> I'm thinking about the END->NVT stage of the process here, rather than
>>>>> the NVT->TCTX stage.
>>>>>
>>>>> Oh, also, you're using "VP" here which IIUC == "NVT".  Can we
>>>>> standardize on one, please.
>>>>
>>>> VP is used in Linux/KVM Linux/Native and skiboot. Yes. it's a mess. 
>>>> Let's have consistent naming in QEMU and use NVT. 
>>>
>>> Right.  And to cover any inevitable missed ones is why I'd like to see
>>> a cheatsheet giving both terms in the header comments somewhere.
>>
>> yes. I have added a list of names in xive.h. 
> 
> Great.  Oh BTW - this is getting big enough, that I wonder if it makes
> sense to create a hw/intc/xive subdir to put things in, then splitting
> IVSE, IVRE, IVPE related code into separate .c files (I'd still expect
> a common .h though).

Here is what I have for now, for the new files only :

   190 include/hw/ppc/xive_regs.h
   343 include/hw/ppc/xive.h
    78 include/hw/ppc/spapr_xive.h
  1453 hw/intc/spapr_xive.c (sPAPRXive + hcalls)
  1678 hw/intc/xive.c       (XiveSource, XiveRouter, XiveENDSource, XiveTCTX, 
                             END helpers)
   864 hw/intc/spapr_xive_kvm.c
  4606 total

I am putting the KVM export definitions in spapr_xive.h and xive.h but all 
the code is in spapr_xive_kvm.c.

So you would rather have something like :
 
   include/hw/ppc/xive_regs.h
   include/hw/ppc/xive.h
   include/hw/ppc/spapr_xive.h

   hw/intc/xive/spapr.c 
   hw/intc/xive/spapr_kvm.c 
   hw/intc/xive/source.c 
   hw/intc/xive/router.c 
   hw/intc/xive/presenter.c 
   hw/intc/xive/tcontext.c 

>> I was wondering if I should put the diagram below somewhere in a .h file 
>> or under doc/specs/.
> 
> I'd prefer it in the .h file.

OK. will do in  xive.h
 
>>
>> Thanks,
>>
>> C.  
>>
>>
>> = XIVE =================================================================
>>
>> The POWER9 processor comes with a new interrupt controller, called
>> XIVE as "eXternal Interrupt Virtualization Engine".
>>
>>
>> * Overall architecture
>>
>>
>>              XIVE Interrupt Controller
>>              +------------------------------------+      IPIs
>>              | +---------+ +---------+ +--------+ |    +-------+
>>              | |VC       | |CQ       | |PC      |----> | CORES |
>>              | |     esb | |         | |        |----> |       |
>>              | |     eas | |  Bridge | |   tctx |----> |       |
>>              | |SC   end | |         | |    nvt | |    |       |
>>  +------+    | +---------+ +----+----+ +--------+ |    +-+-+-+-+
>>  | RAM  |    +------------------|-----------------+      | | |
>>  |      |                       |                        | | |
>>  |      |                       |                        | | |
>>  |      |  +--------------------v------------------------v-v-v--+    other
>>  |      <--+                     Power Bus                      +--> chips
>>  |  esb |  +---------+-----------------------+------------------+
>>  |  eas |            |                       |
>>  |  end |        +---|-----+                 |
>>  |  nvt |       +----+----+|            +----+----+
>>  +------+       |SC       ||            |SC       |
>>                 |         ||            |         |
>>                 | PQ-bits ||            | PQ-bits |
>>                 | local   |+            |  in VC  |
>>                 +---------+             +---------+
>>                    PCIe                 NX,NPU,CAPI
>>
>>                   SC: Source Controller (aka. IVSE)
>>                   VC: Virtualization Controller (aka. IVRE)
>>                   PC: Presentation Controller (aka. IVPE)
>>                   CQ: Common Queue (Bridge)
>>
>>              PQ-bits: 2 bits source state machine (P:pending Q:queued)
>>                  esb: Event State Buffer (Array of PQ bits in an IVSE)
>>                  eas: Event Assignment Structure
>>                  end: Event Notification Descriptor
>>                  nvt: Notification Virtual Target
>>                 tctx: Thread interrupt Context
>>
>>
>> The XIVE IC is composed of three sub-engines :
>>
>>   - Interrupt Virtualization Source Engine (IVSE), or Source
>>     Controller (SC). These are found in PCI PHBs, in the PSI host
>>     bridge controller, but also inside the main controller for the
>>     core IPIs and other sub-chips (NX, CAP, NPU) of the
>>     chip/processor. They are configured to feed the IVRE with events.
>>
>>   - Interrupt Virtualization Routing Engine (IVRE) or Virtualization
>>     Controller (VC). Its job is to match an event source with an Event
>>     Notification Descriptor (END).
>>
>>   - Interrupt Virtualization Presentation Engine (IVPE) or Presentation
>>     Controller (PC). It maintains the interrupt context state of each
>>     thread and handles the delivery of the external exception to the
>>     thread.
>>
>>
>> * XIVE internal tables
>>
>> Each of the sub-engines uses a set of tables to redirect exceptions
>> from event sources to CPU threads.
>>
>>                                           +-------+
>>   User or OS                              |  EQ   |
>>       or                          +------>|entries|
>>   Hypervisor                      |       |  ..   |
>>     Memory                        |       +-------+
>>                                   |           ^
>>                                   |           |
>>              +-------------------------------------------------+
>>                                   |           |
>>   Hypervisor      +------+    +---+--+    +---+--+   +------+
>>     Memory        | ESB  |    | EAT  |    | ENDT |   | NVTT |
>>    (skiboot)      +----+-+    +----+-+    +----+-+   +------+
>>                     ^  |        ^  |        ^  |       ^
>>                     |  |        |  |        |  |       |
>>              +-------------------------------------------------+
>>                     |  |        |  |        |  |       |
>>                     |  |        |  |        |  |       |
>>                +----|--|--------|--|--------|--|-+   +-|-----+    +------+
>>                |    |  |        |  |        |  | |   | | tctx|    |Thread|
>>    IPI or   ---+    +  v        +  v        +  v |---| +  .. |----->     |
>>   HW events    |                                 |   |       |    |      |
>>                |             IVRE                |   | IVPE  |    +------+
>>                +---------------------------------+   +-------+
>>             
>>
>>
>> The IVSE have a 2-bits, P for pending and Q for queued, state machine
>> for each source that allows events to be triggered. They are stored in
>> an array, the Event State Buffer (ESB) and controlled by MMIOs.
>>
>> If the event is let through, the IVRE looks up in the Event Assignment
>> Structure (EAS) table for an Event Notification Descriptor (END)
>> configured for the source. Each Event Notification Descriptor defines
>> a notification path to a CPU and an in-memory Event Queue, in which
>> will be pushed an EQ data for the OS to pull.
>>
>> The IVPE determines if a Notification Virtual Target (NVT) can handle
>> the event by scanning the thread contexts of the VPs dispatched on the
>> processor HW threads. It maintains the interrupt context state of each
>> thread in a NVT table.
>>
>