Re: [PATCH 00/18] hw/riscv: Add Microchip PolarFire SoC Icicle Kit board support

[Cyril.Jean]

On 8/19/20 2:34 AM, Bin Meng wrote:
> EXTERNAL EMAIL: Do not click links or open attachments unless you know the 
> content is safe
>
> On Tue, Aug 18, 2020 at 9:55 PM Anup Patel <anup@brainfault.org> wrote:
>> On Tue, Aug 18, 2020 at 6:39 PM <Cyril.Jean@microchip.com> wrote:
>>> On 8/18/20 7:17 AM, Anup Patel wrote:
>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know the 
>>>> content is safe
>>>>
>>>> On Tue, Aug 18, 2020 at 1:23 AM <Cyril.Jean@microchip.com> wrote:
>>>>> On 8/17/20 8:28 PM, Alistair Francis wrote:
>>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know 
>>>>>> the content is safe
>>>>>>
>>>>>> On Mon, Aug 17, 2020 at 11:12 AM via <qemu-devel@nongnu.org> wrote:
>>>>>>> Hi Anup,
>>>>>>>
>>>>>>> On 8/17/20 11:30 AM, Bin Meng wrote:
>>>>>>>> EXTERNAL EMAIL: Do not click links or open attachments unless you know 
>>>>>>>> the content is safe
>>>>>>>>
>>>>>>>> Hi Anup,
>>>>>>>>
>>>>>>>> On Sat, Aug 15, 2020 at 1:44 AM Anup Patel <anup@brainfault.org> wrote:
>>>>>>>>> On Fri, Aug 14, 2020 at 10:12 PM Bin Meng <bmeng.cn@gmail.com> wrote:
>>>>>>>>>> From: Bin Meng <bin.meng@windriver.com>
>>>>>>>>>>
>>>>>>>>>> This adds support for Microchip PolarFire SoC Icicle Kit board.
>>>>>>>>>> The Icicle Kit board integrates a PolarFire SoC, with one SiFive's
>>>>>>>>>> E51 plus four U54 cores and many on-chip peripherals and an FPGA.
>>>>>>>>> Nice Work !!! This is very helpful.
>>>>>>>> Thanks!
>>>>>>>>
>>>>>>>>> The Microchip HSS is quite convoluted. It has:
>>>>>>>>> 1. DDR Init
>>>>>>>>> 2. Boot device support
>>>>>>>>> 3. SBI support using OpenSBI as library
>>>>>>>>> 4. Simple TEE support
>>>>>>>>>
>>>>>>>>> I think point 1) and 2) above should be part of U-Boot SPL.
>>>>>>>>> The point 3) can be OpenSBI FW_DYNAMIC.
>>>>>>>>>
>>>>>>>>> Lastly,for point 4), we are working on a new OpenSBI feature using
>>>>>>>>> which we can run independent Secure OS and Non-Secure OS using
>>>>>>>>> U-Boot_SPL+OpenSBI (for both SiFive Unleashed and Microchip
>>>>>>>>> PolarFire).
>>>>>>>>>
>>>>>>>>> Do you have plans for adding U-Boot SPL support for this board ??
>>>>>>>> + Cyril Jean from Microchip
>>>>>>>>
>>>>>>>> I will have to leave this question to Cyril to comment.
>>>>>>>>
>>>>>>> I currently do not have a plan to support U-Boot SPL. The idea of the
>>>>>>> HSS is to contain all the silicon specific initialization and
>>>>>>> configuration code within the HSS before jumping to U-Boot S-mode. I
>>>>>>> would rather keep all this within the HSS for the time being. I would
>>>>>>> wait until we reach production silicon before attempting to move this to
>>>>>>> U-Boot SPL as the HSS is likely to contain some opaque silicon related
>>>>>>> changes for another while.
>>>>>> That is unfortunate, a lot of work has gone into making the boot flow
>>>>>> simple and easy to use.
>>>>>>
>>>>>> QEMU now includes OpenSBI by default to make it easy for users to boot
>>>>>> Linux. The Icicle Kit board is now the most difficult QEMU board to
>>>>>> boot Linux on. Not to mention it makes it hard to impossible to
>>>>>> support it in standard tool flows such as meta-riscv.
>>>>>>
>>>>>> Alistair
>>>>> If it is such a problem we can add a U-Boot SPL stage and the HSS can be
>>>>> treated as standard SoC ROM code.
>>>> It's not mandatory for U-Boot SPL to have stable DRAM calibration settings
>>>> from the start itself. The initial U-Boot SPL support for most
>>>> platforms (accross
>>>> architectures) usually include basic working DRAM calibration settings 
>>>> which
>>>> is later on updated with separate patches. Also, we don't need all U-Boot
>>>> drivers to be upstreamed in one-go as this can be done in phases.
>>>>
>>>> The advantage we have with PolarFire SoC Icicle board is that we already
>>>> have a U-Boot S-mode. and we believe the OpenSBI generic platform will
>>>> work fine for PolarFire SoC Icicle board so the only thing missing right 
>>>> now
>>>> is the U-Boot SPL support for OpenSource boot-flow.
>>>>
>>>> It will certainly accelerate open-source development if we have boot-flow
>>>> U-Boot_SPL => OpenSBI (FW_DYNAMIC) => U-Boot_S-mode working
>>>> on PolarFire SoC Icicle board. Initially, we just need DRAM, SD/eMMC,
>>>> and Serial port support for U-Boot SPL and U-Boot S-mode. Later on,
>>>> more patches can add ethernet and other booting device drivers to U-Boot.
>>>>
>>>> Regarding security services of HSS, we are working on a OpenSBI
>>>> feature which will allow HSS security services to run as independent
>>>> binary in M-mode (not linked to OpenSBI) and OpenSBI FW_DYNAMIC
>>>> will be a separate binary acting as a secure monitor.
>>>>
>>>> Regards,
>>>> Anup
>>> What I have in mind is that the external memory will be up and running
>>> by the time we get to U-Boot SPL. In the case of PolarFire SoC the ROM
>>> code equivalent brings up the DDR memory interface so we do not need to
>>> worry about this as part of U-Boot.
>> Keeping DRAM configuration as part of a separate ROM booting stage prior
>> to the U-Boot SPL sounds good to me. This will lead to following boot-flow:
>>
>> ROM/HSS (M-mode) => U-Boot SPL (M-mode) => OpenSBI (M-mode) => U-Boot S-mode
> If we want to keep the HSS, meaning that DDR memory is already
> initialized, then there is no need to create an additional step of
> U-Boot SPL phase, because there is no size limitation any more.
>
> Right now, the boot workflow on PolarFire is:
>
> HSS + OpenSBI (as a library) (M-mode) => U-Boot S-mode
>
> It's just not the canonical way because of the HSS and OpenSBI as a library.
>
> A canonical way should be:
>
> U-Boot SPL (M-mode) => OpenSBI dynamic (M-mode) => U-Boot S-mode
>
> So we should be aiming to replace HSS with U-Boot SPL.
>
>>> Sounds to me the component that needs to be upstreamed next is the eMMC
>>> support so that it can be used as part of U-Boot SPL. Correct?
>> Maybe as a PHASE1 patch series for PolarFire U-Boot support can
>> target the following:
>> 1. Minimal U-Boot board support for PolarFire IcIcle Board with
>> single defconfig to build both U-Boot SPL and U-Boot S-mode
>> 2. Serial port driver (probably re-use existing driver)
>> 3. SD and eMMC driver
>>
>> Supporting SD booting is highly desirable for PHASE1. At least,
>> U-Boot SPL, OpenSBI, U-Boot S-mode, and Linux should come
>> as separate images from SD card. The ROM/HSS can reside and
>> boot from on-board flash/eMMC.
>>
>> Above is my suggestion based on experience with SiFive Unleashed.
>>
>> The general idea behind OpenSource boot-flow is to have each
>> booting stage as a separate binary so that users can selectively
>> upgrade a particular booting stage without re-compiling/re-flashing
>> other booting stages.
>>
>> Various distros and yocto already support building most of the above
>> booting stages (U-Boot, OpenSBI, Linux, etc) so we can reuse a lot
>> of existing work. Eventually, I am confident we will endup moving
>> more stuff from ROM/HSS to U-Boot SPL for ease in maintenance.
> Regards,
> Bin

I am OK with supporting U-Boot SPL as a community boot flow but the HSS 
is not going to disappear completely for many PolarFire SoC deployments. 
We need to have the HSS running on the monitor core after the system has 
booted for various reasons. Some of the current HSS functionalities can 
be migrated at a later time.

The HSS was designed to easily enable and disable features through 
Kconfig which should allow to assign features to different boot stages 
depending on the overall system deployment. Keeping the HSS as ROM code 
capable of starting various early boot stages in either M mode or S mode 
allows us (Microchip engineering and non pure Linux users) to mix 
different operating systems on the same platform and more easily debug 
hardware bring-up.

What I am trying to get at is that we need to balance the QEMU ease of 
use with the ability to bring-up and debug actual hardware. My preferred 
approach is to keep all the hardware bring up in one place, the HSS, so 
that you do not have to worry about obscure silicon idiosyncrasies once 
you reach the open source boot flow. Silicon/board bring up 
steps/features can be brought out of the HSS into the open source boot 
stages once these steps are mature.

PolarFire SoC is the third family of SoC FPGA I have worked on. So I 
have a fair idea of the amount of pain in front of me to bring the 
entire device family to market :-) We decided early on in the 
architecture of PolarFire SoC that we would make all code executing on 
the RISC-V processors open source and user modifiable. This includes all 
code typically hidden away in your typical SoC ROM code. This is why 
PolarFire SoC has on-chip flash memory used to store the HSS and why the 
HSS should be thought as including ROM code. The current Icicle Kit uses 
engineering samples from the first device in the PolarFire SoC family. 
So we are really at the beginning of the process of validating very 
early boot steps on a significant number of devices. I do not want to 
expose the open source community to the pain of bringing up a family of 
devices because of the decision to open source all code. So I am asking 
you to not rush in trying to migrate components from the HSS to a later 
boot stage just yet. It can be done but based on experience I think that 
now is not the right time yet.

Regards,

Cyril.