Re: Collecting GRUB 2.06 test results

[Glenn Washburn]

On Thu, 11 Feb 2021 01:18:38 +0100
John Paul Adrian Glaubitz <glaubitz@physik.fu-berlin.de> wrote:

> Hi Glenn!
> 
> On 2/11/21 12:57 AM, Glenn Washburn wrote:
> > Ok, I'm glad you're running the tests on native hardware and in that
> > case I'm not surprised most of the qemu tests aren't running
> > because I assume that qemu is not on most of the non x86 platforms.
> > Perhaps we should have the tests show as SKIPPED when the
> > appropriate qemu binary can not be found.
> 
> OK. I wasn't aware of that.

Looks like I might have been wrong about my assumption that qemu is not
on our other supported architectures. If I'm reading this table
correctly, https://wiki.qemu.org/Documentation/Platforms, it looks like
qemu should be able to virtualize any supported platform for arm,
arm64, mips, powerpc, and sparc. I would guess that debian does build
all the supported qemu architectures for those platforms. That way you
could say run the make check tests on a powerpc host for the arm64
target. This would be a way of testing that GRUB cross-compiled on
powerpc for arm64 could in fact boot on an arm64 machine, and more
generally testing every host + target combinations that we support.

> >>> I'd like to note that it appears that these tests do not perform
> >>> most of the various filesystem tests.
> >>
> >> I ran the tests with "make check" (or "make test"), I did not make
> >> any modifications. If some tests were not run that's because GRUB
> >> does not enable them on the target in question.
> > 
> > That's technically true, but not the whole story. GRUB is not
> > enabling the filesystem tests because you don't have your target
> > environment setup such that the tests will be enabled. The
> > filesystem tests require filesystem tools (eg. e2fsprogs for ext*
> > fs tests). You might try installing the prerequisites needed by the
> > tests, and then seeing if they run.
> 
> I can re-run them with the fsprogs installed. Does that just concern
> e2fsprogs or other filesystems as well?

There's a lot more than e2fsprogs to get all the filesystems being
tested. I'm putting the final touches on a CI system I've made for GRUB
and here's the list of ubuntu packages I install for running the tests.
I think they should be the same for debian.

qemu-system
ovmf
openbios-ppc
openbios-sparc
wamerican
xorriso
locales
language-pack-en
mtools
tar
cpio
gzip
lzop
xz-utils
parted
util-linux
squashfs-tools
zfsutils-linux
dosfstools
exfat-utils
ntfs-3g
e2fsprogs
btrfs-progs
xfsprogs
hfsprogs
jfsutils
reiserfsprogs
udftools
nilfs-tools
f2fs-tools
genromfs
attr

> >> You can see that by looking at the configure.ac. It's basically all
> >> targets that build more than just the utilities.
> > 
> > Great, that's what I was thinking. I notice that it appears you're
> > not testing every supported target, just one target from each
> > architecture. For instance, you're not testing and i386 build of
> > the ieee1275 or efi platforms. 
> 
> Well, if ieee1275 or efi should be tested on i386, I think the
> testsuite should enable these tests by default. Although I'm not sure
> there is x86 hardware which uses OpenFirmware, is there?

GRUB is built for a particular target, which is an architecture,
platform combination. The test suite only runs for the built target. So
if you're not building for multiple targets on the same architecture,
then you're not testing all targets for that architecture. I'm not
familiar with the hardware of most of the targets GRUB supports, but I
do believe that i386-ieee1275 is supported.

> > Since you're not using qemu on the non-x86 platforms, I'm not sure
> > if testing the various platforms of other architectures would be
> > worth while (eg. arm-coreboot and arm-efi). Perhaps someone more
> > knowledgeable than I can weight in.
> 
> I can also test with QEMU installed if desired for certain
> architectures.

I think that would make for better quality test output. On ubuntu I
install qemu-system and get all the right binaries.

> Maybe you could make a list of target/architecture combinations you
> want me to test.

Take a look at the .travis.yml file in the matrix section. You'll see
the variable GRUB_TARGETS defined multiple times, once for each
architecture. GRUB_TARGETS is itself a list of targets. I believe, we
should ideally testing all those targets. In practice, I've had a hard
time getting quite a few of them to work (tests to run properly),
including i386-ieee1275 (haven't found a build of open firmware for
i386 qemu). 

Glenn