Re: [Qemu-devel] [PATCH 24/58] PPC: E500: Add PV spinning code

[Alexander Graf]

On 27.09.2011, at 19:20, Blue Swirl wrote:

> On Tue, Sep 27, 2011 at 5:03 PM, Alexander Graf <address@hidden> wrote:
>> 
>> On 27.09.2011, at 18:53, Blue Swirl wrote:
>> 
>>> On Tue, Sep 27, 2011 at 3:59 PM, Alexander Graf <address@hidden> wrote:
>>>> 
>>>> On 27.09.2011, at 17:50, Blue Swirl wrote:
>>>> 
>>>>> On Mon, Sep 26, 2011 at 11:19 PM, Scott Wood <address@hidden> wrote:
>>>>>> On 09/24/2011 05:00 AM, Alexander Graf wrote:
>>>>>>> On 24.09.2011, at 10:44, Blue Swirl wrote:
>>>>>>>> On Sat, Sep 24, 2011 at 8:03 AM, Alexander Graf <address@hidden> wrote:
>>>>>>>>> On 24.09.2011, at 09:41, Blue Swirl wrote:
>>>>>>>>>> On Mon, Sep 19, 2011 at 4:12 PM, Scott Wood <address@hidden> wrote:
>>>>>>>>>>> The goal with the spin table stuff, suboptimal as it is, was 
>>>>>>>>>>> something
>>>>>>>>>>> that would work on any powerpc implementation.  Other
>>>>>>>>>>> implementation-specific release mechanisms are allowed, and are
>>>>>>>>>>> indicated by a property in the cpu node, but only if the loader 
>>>>>>>>>>> knows
>>>>>>>>>>> that the OS supports it.
>>>>>>>>>>> 
>>>>>>>>>>>> IIUC the spec that includes these bits is not finalized yet. It is 
>>>>>>>>>>>> however in use on all u-boot versions for e500 that I'm aware of 
>>>>>>>>>>>> and the method Linux uses to bring up secondary CPUs.
>>>>>>>>>>> 
>>>>>>>>>>> It's in ePAPR 1.0, which has been out for a while now.  ePAPR 1.1 
>>>>>>>>>>> was
>>>>>>>>>>> just released which clarifies some things such as WIMG.
>>>>>>>>>>> 
>>>>>>>>>>>> Stuart / Scott, do you have any pointers to documentation where 
>>>>>>>>>>>> the spinning is explained?
>>>>>>>>>>> 
>>>>>>>>>>> https://www.power.org/resources/downloads/Power_ePAPR_APPROVED_v1.1.pdf
>>>>>>>>>> 
>>>>>>>>>> Chapter 5.5.2 describes the table. This is actually an interface
>>>>>>>>>> between OS and Open Firmware, obviously there can't be a real 
>>>>>>>>>> hardware
>>>>>>>>>> device that magically loads r3 etc.
>>>>>> 
>>>>>> Not Open Firmware, but rather an ePAPR-compliant loader.
>>>>> 
>>>>> 'boot program to client program interface definition'.
>>>>> 
>>>>>>>>>> The device method would break abstraction layers,
>>>>>> 
>>>>>> Which abstraction layers?
>>>>> 
>>>>> QEMU system emulation emulates hardware, not software. Hardware
>>>>> devices don't touch CPU registers.
>>>> 
>>>> The great part about this emulated device is that it's basically guest 
>>>> software running in host context. To the guest, it's not a device in the 
>>>> ordinary sense, such as vmport, but rather the same as software running on 
>>>> another core, just that the other core isn't running any software.
>>>> 
>>>> Sure, if you consider this a device, it does break abstraction layers. 
>>>> Just consider it as host running guest code, then it makes sense :).
>>>> 
>>>>> 
>>>>>>>>>> it's much like
>>>>>>>>>> vmport stuff in x86. Using a hypercall would be a small improvement.
>>>>>>>>>> Instead it should be possible to implement a small boot ROM which 
>>>>>>>>>> puts
>>>>>>>>>> the secondary CPUs into managed halt state without spinning, then the
>>>>>>>>>> boot CPU could send an IPI to a halted CPU to wake them up based on
>>>>>>>>>> the spin table, just like real HW would do.
>>>>>> 
>>>>>> The spin table, with no IPI or halt state, is what real HW does (or
>>>>>> rather, what software does on real HW) today.  It's ugly and inefficient
>>>>>> but it should work everywhere.  Anything else would be dependent on a
>>>>>> specific HW implementation.
>>>>> 
>>>>> Yes. Hardware doesn't ever implement the spin table.
>>>>> 
>>>>>>>>>> On Sparc32 OpenBIOS this
>>>>>>>>>> is something like a few lines of ASM on both sides.
>>>>>>>>> 
>>>>>>>>> That sounds pretty close to what I had implemented in v1. Back then 
>>>>>>>>> the only comment was to do it using this method from Scott.
>>>>>> 
>>>>>> I had some comments on the actual v1 implementation as well. :-)
>>>>>> 
>>>>>>>>> So we have the choice between having code inside the guest that
>>>>>>>>> spins, maybe even only checks every x ms, by programming a timer,
>>>>>>>>> or we can try to make an event out of the memory write. V1 was
>>>>>>>>> the former, v2 (this one) is the latter. This version performs a
>>>>>>>>> lot better and is easier to understand.
>>>>>>>> 
>>>>>>>> The abstraction layers should not be broken lightly, I suppose some
>>>>>>>> performance or laziness^Wlocal optimization reasons were behind vmport
>>>>>>>> design too. The ideal way to solve this could be to detect a spinning
>>>>>>>> CPU and optimize that for all architectures, that could be tricky
>>>>>>>> though (if a CPU remains in the same TB for extended periods, inspect
>>>>>>>> the TB: if it performs a loop with a single load instruction, replace
>>>>>>>> the load by a special wait operation for any memory stores to that
>>>>>>>> page).
>>>>>> 
>>>>>> How's that going to work with KVM?
>>>>>> 
>>>>>>> In fact, the whole kernel loading way we go today is pretty much
>>>>>>> wrong. We should rather do it similar to OpenBIOS where firmware
>>>>>>> always loads and then pulls the kernel from QEMU using a PV
>>>>>>> interface. At that point, we would have to implement such an
>>>>>>> optimization as you suggest. Or implement a hypercall :).
>>>>>> 
>>>>>> I think the current approach is more usable for most purposes.  If you
>>>>>> start U-Boot instead of a kernel, how do pass information on from the
>>>>>> user (kernel, rfs, etc)?  Require the user to create flash images[1]?
>>>>> 
>>>>> No, for example OpenBIOS gets the kernel command line from fw_cfg device.
>>>>> 
>>>>>> Maybe that's a useful mode of operation in some cases, but I don't think
>>>>>> we should be slavishly bound to it.  Think of the current approach as
>>>>>> something between whole-system and userspace emulation.
>>>>> 
>>>>> This is similar to ARM, M68k and Xtensa semi-hosting mode, but not at
>>>>> kernel level but lower. Perhaps this mode should be enabled with
>>>>> -semihosting flag or a new flag. Then the bare metal version could be
>>>>> run without the flag.
>>>> 
>>>> and then we'd have 2 implementations for running in system emulation mode 
>>>> and need to maintain both. I don't think that scales very well.
>>> 
>>> No, but such hacks are not common.
>>> 
>>>>> 
>>>>>> Where does the device tree come from?  How do you tell the guest about
>>>>>> what devices it has, especially in virtualization scenarios with non-PCI
>>>>>> passthrough devices, or custom qdev instantiations?
>>>>>> 
>>>>>>> But at least we'd always be running the same guest software stack.
>>>>>> 
>>>>>> No we wouldn't.  Any U-Boot that runs under QEMU would have to be
>>>>>> heavily modified, unless we want to implement a ton of random device
>>>>>> emulation, at least one extra memory translation layer (LAWs, localbus
>>>>>> windows, CCSRBAR, and such), hacks to allow locked cache lines to
>>>>>> operate despite a lack of backing store, etc.
>>>>> 
>>>>> I'd say HW emulation business as usual. Now with the new memory API,
>>>>> it should be possible to emulate the caches with line locking and TLBs
>>>>> etc., this was not previously possible. IIRC implementing locked cache
>>>>> lines would allow x86 to boot unmodified coreboot.
>>>> 
>>>> So how would you emulate cache lines with line locking on KVM?
>>> 
>>> The cache would be a MMIO device which registers to handle all memory
>>> space. Configuring the cache controller changes how the device
>>> operates. Put this device between CPU and memory and other devices.
>>> Performance would probably be horrible, so CPU should disable the
>>> device automatically after some time.
>> 
>> So how would you execute code on this region then? :)
> 
> Easy, fix QEMU to allow executing from MMIO. (Yeah, I forgot about that).

It's not quite as easy to fix KVM to do the same though unfortunately. We'd 
have to either implement a full instruction emulator in the kernel (x86 style) 
or transfer all state from KVM into QEMU to execute it there (hell breaks 
loose). Both alternatives are not exactly appealing.

> 
>>> 
>>>> However, we already have a number of hacks in SeaBIOS to run in QEMU, so I 
>>>> don't see an issue in adding a few here and there in u-boot. The memory 
>>>> pressure is a real issue though. I'm not sure how we'd manage that one. 
>>>> Maybe we could try and reuse the host u-boot binary? heh
>>> 
>>> I don't think SeaBIOS breaks layering except for fw_cfg.
>> 
>> I'm not saying we're breaking layering there. I'm saying that changing 
>> u-boot is not so bad, since it's the same as we do with SeaBIOS. It was an 
>> argument in favor of your position.
> 
> Never mind then ;-)
> 
>>> For extremely
>>> memory limited situation, perhaps QEMU (or Native KVM Tool for lean
>>> and mean version) could be run without glibc, inside kernel or even
>>> interfacing directly with the hypervisor. I'd also continue making it
>>> possible to disable building unused devices and features.
>> 
>> I'm pretty sure you're not the only one with that goal ;).
> 
> Great, let's do it.

VGA comes first :)


Alex