Re: strange grub error on ARM64

[Michael Chang]

On Thu, Mar 09, 2023 at 02:10:54AM -0800, Atish Patra wrote:
> On Mon, Mar 6, 2023 at 7:21 PM Michael Chang <mchang@suse.com> wrote:
> >
> > On Mon, Mar 06, 2023 at 04:22:53PM -0800, Atish Patra wrote:
> > > On Mon, Mar 6, 2023 at 1:25 PM Fabian Vogt <fvogt@suse.de> wrote:
> > > >
> > > > Hi,
> > > >
> > > > Am Montag, 6. März 2023, 21:59:23 CET schrieb Daniel Kiper:
> > > > > Adding Gary and Michael...
> > > > >
> > > > > Gary, Michael, could you help with this? Or at least forward this 
> > > > > email
> > > > > to folks who may help...
> > > > >
> > > > > Daniel
> > > > >
> > > > > On Mon, Mar 06, 2023 at 10:56:52AM -0800, Atish Patra wrote:
> > > > > > Hi,
> > > > > > I am working on a cross-architecture cleanup series that removes the
> > > > > > arch specific headers from RISC-V, ARM, ARM64 and unifies the linux
> > > > > > loader on ARM64 & Linux.
> > > > > >
> > > > > > The v7 version can be found here.
> > > > > > https://lists.gnu.org/archive/html/grub-devel/2023-02/msg00046.html
> > > > > >
> > > > > > It had a few bugs which are fixed and can be found here.
> > > > > >
> > > > > > Here are the patches
> > > > > > https://github.com/atishp04/grub/commits/riscv_uefi_v8
> > > > > >
> > > > > > Anybody with ARM64 and/or test setup: Can you please do a quick test
> > > > > > to see if you see similar or any other failure ?
> > > > > >
> > > > > > I noticed a very strange error during ARM64 testing. I have probably
> > > > > > done something stupid during the grub build
> > > > > > as I can reproduce this with upstream grub as well.
> > > > > >
> > > > > > I was trying to boot an OpenSuse image in qemu virt machine for 
> > > > > > ARM64 as per
> > > > > > the instructions[1]. The stock grub image works fine however my 
> > > > > > custom
> > > > > > build fails to boot
> > > > > > and drops to the grub command line. It seems the grub detects the 
> > > > > > disk
> > > > > > but doesn't read the root partition correctly on my cross-compiled 
> > > > > > image.
> > > > > > It has one additional directory ("@") which I don't see when I mount
> > > > > > the disk. it doesn't list any files as well.
> > > >
> > > > What you see is the expected behaviour with upstream GRUB.
> > > > The btrfs volume contains various subvolumes below @ which are mounted 
> > > > into the
> > > > root file system at various points. The root filesystem itself is at
> > > > @/.snapshots/<number>/snapshot/ and set as the default subvolume. 
> > > > That's what
> > > > is mounted in a running system if you pass default flags. To see the 
> > > > same as
> > > > GRUB, you can do mount -o subvolume=/ /dev/foo3 /mnt.
> > > >
> > >
> > > Thanks!! Thank god I am not crazy as I thought I was ;)
> > >
> > > > Downstream there's a patch (and multiple related ones) that makes GRUB 
> > > > mount
> > > > the default subvolume instead of the root of the filesystem:
> > > > https://build.opensuse.org/package/view_file/openSUSE:Factory/grub2/grub2-btrfs-03-follow_default.patch?expand=1&rev=285
> > > >
> > >
> > > Is this the only patch or I need to pull all the patches on that tree
> > > to make it work ?
> >
> > You'll need other patch so that files not in default subvolume can be
> > accessed through mount at grub's runtime, also to enable automatic setup
> > via the utility grub-(install|mkconfig).
> >
> > >
> > > Is there an easy way to test upstream grub stuff on ARM64 in case I
> > > need to pull more dependent patches to make it work?
> >
> > You can also use rpm from this Open Build Service (OBS) project:
> >
> > https://build.opensuse.org/package/show/home:michael-chang:grub:devel/grub2
> >
> > It is for merging upstream git with SUSE patches rebased regularly per
> > month convenient to catch early build failure, find regression, try out
> > new feature and so on.
> >
> 
> Thanks for pointing that out. But it seems all the patches are on top
> of some older release of grub.
> I am trying to test patches on top of the upstream and some of these
> patches may not apply cleanly.
> 
> I was looking for alternatives with other distros. But every other
> distro probably carries a similar number
> of out-of-tree patches.

The downstream patch may jump to PE entry point directly for verified
image buffer given UEFI LoadImage() will return security violation for
verified signatures by the key only exist in Machine Owner Key (MOK)
keyring.

> Here is the fedora one.
> 
> https://src.fedoraproject.org/rpms/grub2/blob/rawhide/f/grub.patches
> 
> I have a small favor to ask. Is it possible for you to test out the
> changes[1] on the ARM64 platform ?

Sure. I'll do it next week and come back to you for any result.

> You probably have to rebase suse patches on top of upstream first as
> well. But you would probably do it much much much faster than me
> given my limited knowledge about grub & suse.
> 
> [1] https://github.com/atishp04/grub/commits/riscv_uefi_v8

The riscv_uefi would have been riscv_arm64_uefi as long as they are
synchronized in this branch. :)

Thanks,
Michael

> 
> > Thanks,
> > Michael
> >
> > >
> > > > The design of the "Boot into Snapshot" functionality relies on this.
> > > >
> > > > > > ============= Cross compiled grub image =====================
> > > > > > grub> ls -l (hd0,gpt3)/
> > > > > > DIR          20230302080325 @/
> > > > > >
> > > > > > grub> ls -l (hd0,gpt3)/@/
> > > > > > DIR          20230302080326 boot/
> > > > > > DIR         srv/
> > > > > > DIR          20230302080326 usr/
> > > > > > DIR         opt/
> > > > > > DIR          20230302080326 home/
> > > > > > DIR         var/
> > > > > > DIR         tmp/
> > > > > > DIR         root/
> > > > > >
> > > > > > grub> ls -l (hd0,gpt3)/@/boot/
> > > > > > DIR          20230302080326 grub2/
> > > > > >
> > > > > > grub>
> > > > > > ============= Stock grub image from the OpenSuse 
> > > > > > image=====================
> > > > > > grub> ls (hd0,gpt3)/
> > > > > > dev/ run/ boot/ config.partids lib sbin srv/ usr/ opt/ 
> > > > > > config.bootoptions lib64
> > > > > > .snapshots/ bin home/ etc/ var/ tmp/ sys/ mnt/ proc/ root/
> > > > > >
> > > > > > The same build steps works fine for RISC-V for OpenSuse image 
> > > > > > though.
> > > > > > However, I couldn't figure out what it was. Any help is appreciated.
> > > >
> > > > It's possible that it does not use snapshot or not even btrfs.
> > > >
> > > > Cheers,
> > > > Fabian
> > > >
> > > > > > [1] https://en.opensuse.org/openSUSE:AArch64
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > > Atish
> > > > >
> > > > > _______________________________________________
> > > > > Grub-devel mailing list
> > > > > Grub-devel@gnu.org
> > > > > https://lists.gnu.org/mailman/listinfo/grub-devel
> > > >
> > > >
> > >
> > >
> > > --
> > > Regards,
> > > Atish
> 
> 
> 
> -- 
> Regards,
> Atish