Re: [Qemu-devel] [edk2-devel] CPU hotplug using SMM with QEMU+OVMF

[Igor Mammedov]

On Thu, 15 Aug 2019 17:00:16 +0200
Laszlo Ersek <address@hidden> wrote:

> On 08/14/19 16:04, Paolo Bonzini wrote:
> > On 14/08/19 15:20, Yao, Jiewen wrote:  
> >>> - Does this part require a new branch somewhere in the OVMF SEC code?
> >>>   How do we determine whether the CPU executing SEC is BSP or
> >>>   hot-plugged AP?  
> >> [Jiewen] I think this is blocked from hardware perspective, since the 
> >> first instruction.
> >> There are some hardware specific registers can be used to determine if the 
> >> CPU is new added.
> >> I don’t think this must be same as the real hardware.
> >> You are free to invent some registers in device model to be used in OVMF 
> >> hot plug driver.  
> > 
> > Yes, this would be a new operation mode for QEMU, that only applies to
> > hot-plugged CPUs.  In this mode the AP doesn't reply to INIT or SMI, in
> > fact it doesn't reply to anything at all.
> >   
> >>> - How do we tell the hot-plugged AP where to start execution? (I.e. that
> >>>   it should execute code at a particular pflash location.)  
> >> [Jiewen] Same real mode reset vector at FFFF:FFF0.  
> > 
> > You do not need a reset vector or INIT/SIPI/SIPI sequence at all in
> > QEMU.  The AP does not start execution at all when it is unplugged, so
> > no cache-as-RAM etc.
> > 
> > We only need to modify QEMU so that hot-plugged APIs do not reply to
> > INIT/SIPI/SMI.
> >   
> >> I don’t think there is problem for real hardware, who always has CAR.
> >> Can QEMU provide some CPU specific space, such as MMIO region?  
> > 
> > Why is a CPU-specific region needed if every other processor is in SMM
> > and thus trusted.  
> 
> I was going through the steps Jiewen and Yingwen recommended.
> 
> In step (02), the new CPU is expected to set up RAM access. In step
> (03), the new CPU, executing code from flash, is expected to "send board
> message to tell host CPU (GPIO->SCI) -- I am waiting for hot-add
> message." For that action, the new CPU may need a stack (minimally if we
> want to use C function calls).
> 
> Until step (03), there had been no word about any other (= pre-plugged)
> CPUs (more precisely, Jiewen even confirmed "No impact to other
> processors"), so I didn't assume that other CPUs had entered SMM.
> 
> Paolo, I've attempted to read Jiewen's response, and yours, as carefully
> as I can. I'm still very confused. If you have a better understanding,
> could you please write up the 15-step process from the thread starter
> again, with all QEMU customizations applied? Such as, unnecessary steps
> removed, and platform specifics filled in.
> 
> One more comment below:
> 
> >   
> >>>   Does CPU hotplug apply only at the socket level? If the CPU is
> >>>   multi-core, what is responsible for hot-plugging all cores present in
> >>>   the socket?  
> > 
> > I can answer this: the SMM handler would interact with the hotplug
> > controller in the same way that ACPI DSDT does normally.  This supports
> > multiple hotplugs already.
> > 
> > Writes to the hotplug controller from outside SMM would be ignored.
> >   
> >>>> (03) New CPU: (Flash) send board message to tell host CPU (GPIO->SCI)
> >>>>      -- I am waiting for hot-add message.  
> >>>
> >>> Maybe we can simplify this in QEMU by broadcasting an SMI to existent
> >>> processors immediately upon plugging the new CPU.  
> > 
> > The QEMU DSDT could be modified (when secure boot is in effect) to OUT
> > to 0xB2 when hotplug happens.  It could write a well-known value to
> > 0xB2, to be read by an SMI handler in edk2.  
> 
> (My comment below is general, and may not apply to this particular
> situation. I'm too confused to figure that out myself, sorry!)
> 
> I dislike involving QEMU's generated DSDT in anything SMM (even
> injecting the SMI), because the AML interpreter runs in the OS.
> 
> If a malicious OS kernel is a bit too enlightened about the DSDT, it
> could willfully diverge from the process that we design. If QEMU
> broadcast the SMI internally, the guest OS could not interfere with that.
> 
> If the purpose of the SMI is specifically to force all CPUs into SMM
> (and thereby force them into trusted state), then the OS would be
> explicitly counter-interested in carrying out the AML operations from
> QEMU's DSDT.
it shouldn't matter where from management SMI comes if OS won't be able
to actually trigger SMI with un-trusted content at SMBASE on hotplugged 
(parked) CPU.
The worst that could happen is that new cpu will stay parked.

> I'd be OK with an SMM / SMI involvement in QEMU's DSDT if, by diverging
> from that DSDT, the OS kernel could only mess with its own state, and
> not with the firmware's.
> 
> Thanks
> Laszlo
> 
> > 
> >   
> >>>  
> >>>>                                        (NOTE: Host CPU can only  
> >>> send  
> >>>>      instruction in SMM mode. -- The register is SMM only)  
> >>>
> >>> Sorry, I don't follow -- what register are we talking about here, and
> >>> why is the BSP needed to send anything at all? What "instruction" do you
> >>> have in mind?  
> >> [Jiewen] The new CPU does not enable SMI at reset.
> >> At some point of time later, the CPU need enable SMI, right?
> >> The "instruction" here means, the host CPUs need tell to CPU to enable 
> >> SMI.  
> > 
> > Right, this would be a write to the CPU hotplug controller
> >   
> >>>> (04) Host CPU: (OS) get message from board that a new CPU is added.
> >>>>      (GPIO -> SCI)
> >>>>
> >>>> (05) Host CPU: (OS) All CPUs enter SMM (SCI->SWSMI) (NOTE: New CPU
> >>>>      will not enter CPU because SMI is disabled)  
> >>>
> >>> I don't understand the OS involvement here. But, again, perhaps QEMU can
> >>> force all existent CPUs into SMM immediately upon adding the new CPU.  
> >> [Jiewen] OS here means the Host CPU running code in OS environment, not in 
> >> SMM environment.  
> > 
> > See above.
> >   
> >>>> (06) Host CPU: (SMM) Save 38000, Update 38000 -- fill simple SMM
> >>>>      rebase code.
> >>>>
> >>>> (07) Host CPU: (SMM) Send message to New CPU to Enable SMI.  
> >>>
> >>> Aha, so this is the SMM-only register you mention in step (03). Is the
> >>> register specified in the Intel SDM?  
> >> [Jiewen] Right. That is the register to let host CPU tell new CPU to 
> >> enable SMI.
> >> It is platform specific register. Not defined in SDM.
> >> You may invent one in device model.  
> > 
> > See above.
> >   
> >>>> (10) New CPU: (SMM) Response first SMI at 38000, and rebase SMBASE to
> >>>>      TSEG.  
> >>>
> >>> What code does the new CPU execute after it completes step (10)? Does it
> >>> halt?  
> >>
> >> [Jiewen] The new CPU exits SMM and return to original place - where it is
> >> interrupted to enter SMM - running code on the flash.  
> > 
> > So in our case we'd need an INIT/SIPI/SIPI sequence between (06) and (07).
> >   
> >>>> (11) Host CPU: (SMM) Restore 38000.  
> >>>
> >>> These steps (i.e., (06) through (11)) don't appear RAS-specific. The
> >>> only platform-specific feature seems to be SMI masking register, which
> >>> could be extracted into a new SmmCpuFeaturesLib API.
> >>>
> >>> Thus, would you please consider open sourcing firmware code for steps
> >>> (06) through (11)?
> >>>
> >>> Alternatively -- and in particular because the stack for step (01)
> >>> concerns me --, we could approach this from a high-level, functional
> >>> perspective. The states that really matter are the relocated SMBASE for
> >>> the new CPU, and the state of the full system, right at the end of step
> >>> (11).
> >>>
> >>> When the SMM setup quiesces during normal firmware boot, OVMF could
> >>> use
> >>> existent (finalized) SMBASE infomation to *pre-program* some virtual
> >>> QEMU hardware, with such state that would be expected, as "final" state,
> >>> of any new hotplugged CPU. Afterwards, if / when the hotplug actually
> >>> happens, QEMU could blanket-apply this state to the new CPU, and
> >>> broadcast a hardware SMI to all CPUs except the new one.  
> > 
> > I'd rather avoid this and stay as close as possible to real hardware.
> > 
> > Paolo
> > 
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Groups.io Links: You receive all messages sent to this group.
> > 
> > View/Reply Online (#45608): https://edk2.groups.io/g/devel/message/45608
> > Mute This Topic: https://groups.io/mt/32852911/1721875
> > Group Owner: address@hidden
> > Unsubscribe: https://edk2.groups.io/g/devel/unsub  [address@hidden]
> > -=-=-=-=-=-=-=-=-=-=-=-
> >   
>