Re: Grub2.02 installed with Pythin 3, boots Xen Dom0 but not Xen DomU-s

[Vladimir 'phcoder' Serbinenko]

Pygrub is not part of grub project. 

On Sun, 21 Jul 2019, 12:58 Kevin Buckley, <address@hidden> wrote:

Hi there,

this may well fall into a "Grub2 should not be run under Python3"
basket, hence not submitting a bug at savannah, however the fact
that my Grub appears to allow me to boot my Dom0 system, but not
boot my Xen DomU-s, has me thinking there's more to it than that.


OK, so I am trying to build an LFS (Linux From Scratch) system that
will serve as a basic Xen Dom0.

You can see the way things have been compiled, including my UEFI-aware
Grub, here (Very much a WIP)

  http://youvegotbuckleys.org.nz/LFS/LFS-BOOK.html


Suffice it to say that the LFS Grub boots either of the two
UEFI enties (non-Xen and Dom0) I have so as to bring up the
system.


I also have a couple of VBD-backed DomU-s that I can bring up,
using the pygrub from either an Ubuntu 1404, or a Centos 6.10
with Xen4Centos, Dom0.

When I come to try and have my LFS Xen's pygrub boot the VBD-backed
DomU, I get errors of the form


# cat /mnt/var/log/xen/bootloader.4.log
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
Traceback (most recent call last):
  File "/usr/lib/xen/bin/pygrub", line 928, in <module>
    raise RuntimeError("Unable to find partition containing kernel")
RuntimeError: Unable to find partition containing kernel

I originally thought that this might be down to my using a Xen
source from the Xen Git repo that was in beyond 4.12 (which I
needed so to get a Xen that was happy with just Python3. See
 https://lists.xenproject.org/archives/html/xen-devel/2019-04/msg00996.html
) and had prhaps had tripped over a regression, as that error used
to be a problem with Xen (if you go back far enough).


However I have since tracked down my problem to the pygrub, in
that if I point pygrub to the start of the partition in the VBD
that has the grub.cfg in it, I see the following failure to parse
the config file:


bash-5.0# /usr/lib/xen/bin/pygrub --debug --offset=1048576
--list-entries /dev/vg_xen_vbds/lv_4g_02
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
Traceback (most recent call last):
  File "/usr/lib/xen/bin/pygrub", line 907, in <module>
    chosencfg = run_grub(file, entry, fs, incfg["args"])
  File "/usr/lib/xen/bin/pygrub", line 625, in run_grub
    g = Grub(file, fs)
  File "/usr/lib/xen/bin/pygrub", line 249, in __init__
    self.read_config(file, fs)
  File "/usr/lib/xen/bin/pygrub", line 460, in read_config
    self.cf.parse(buf)
  File "/usr/lib/python3.7/site-packages/grub/GrubConf.py", line 376, in parse
    lines = buf.split("\n")
TypeError: a bytes-like object is required, not 'str'


whereas if I do the same with the pygrub from the CentOs6.10/Xen4Centos
environment, I see


/usr/bin/pygrub  --debug --offset=1048576 --list-entries
/dev/vg_xen_vbds/lv_4g_02
title: Ubuntu
  root: None
  kernel: /boot/vmlinuz-4.4.0-31-generic
  args: root=/dev/xvda1 ro  quiet splash $vt_handoff
  initrd: /boot/initrd.img-4.4.0-31-generic
title: Ubuntu, with Linux 4.4.0-31-generic
  root: None
  kernel: /boot/vmlinuz-4.4.0-31-generic
  args: root=UUID=5556a819-ced8-4864-9e7f-73792570703e ro  quiet
splash $vt_handoff
  initrd: /boot/initrd.img-4.4.0-31-generic
title: Ubuntu, with Linux 4.4.0-31-generic (recovery mode)
  root: None
  kernel: /boot/vmlinuz-4.4.0-31-generic
  args: root=UUID=5556a819-ced8-4864-9e7f-73792570703e ro recovery nomodeset
  initrd: /boot/initrd.img-4.4.0-31-generic
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
INFO:root:Ignored directive if
INFO:root:Ignored directive load_env
INFO:root:Ignored directive fi
INFO:root:Ignored directive if
WARNING:root:grub2's saved_entry/next_entry not supported
INFO:root:Ignored directive save_env
...


and it goes on and suceeds.

Ok, so then I went hunting around for the cause of the error
I was seeing on the LFS system and got pointed towards the
cause being a Pythin2->3 issue, with the way 3 now differentiates
between strings and bytes.


FWIW, I tried making some changes to the GrubConf.py file but only
seemed to see more worms coming out of each can that I opened. The
first of my attempted changes was to try and decode the buffer but
that seems to have moved me off onto another error again, whilst the
second line of attack, where I tried to leave the buffer as "bytes"
whilst altering the operations on it to be "byte-aware" only
compilacted things further.

The odd things for me is that the same pygrub that can boot my
LFS system doesn't appear to be failing.

I present the chnages that I made and the resulting errors from
my attempts to point pygrub at the partition with the grub.cfg
file on here:

bash-5.0# /usr/lib/xen/bin/pygrub --debug --offset=1048576
--list-entries /dev/vg_xen_vbds/lv_4g_02
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
Traceback (most recent call last):
  File "/usr/lib/xen/bin/pygrub", line 907, in <module>
    chosencfg = run_grub(file, entry, fs, incfg["args"])
  File "/usr/lib/xen/bin/pygrub", line 625, in run_grub
    g = Grub(file, fs)
  File "/usr/lib/xen/bin/pygrub", line 249, in __init__
    self.read_config(file, fs)
  File "/usr/lib/xen/bin/pygrub", line 460, in read_config
    self.cf.parse(buf)
  File "/usr/lib/python3.7/site-packages/grub/GrubConf.py", line 376, in parse
    lines = buf.split("\n")
TypeError: a bytes-like object is required, not 'str'

bash-5.0# vim /usr/lib/python3.7/site-packages/grub/GrubConf.py +376

bash-5.0# diff  GrubConf.py /usr/lib/python3.7/site-packages/grub/GrubConf.py
376c376
<             lines = buf.split("\n")
---
>             lines = buf.decode().split("\n")


bash-5.0# /usr/lib/xen/bin/pygrub --debug --offset=1048576
--list-entries /dev/vg_xen_vbds/lv_4g_02
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
INFO:root:Ignored directive if
INFO:root:Ignored directive load_env
INFO:root:Ignored directive fi
INFO:root:Ignored directive if
WARNING:root:grub2's saved_entry/next_entry not supported
Traceback (most recent call last):
  File "/usr/lib/xen/bin/pygrub", line 907, in <module>
    chosencfg = run_grub(file, entry, fs, incfg["args"])
  File "/usr/lib/xen/bin/pygrub", line 625, in run_grub
    g = Grub(file, fs)
  File "/usr/lib/xen/bin/pygrub", line 249, in __init__
    self.read_config(file, fs)
  File "/usr/lib/xen/bin/pygrub", line 460, in read_config
    self.cf.parse(buf)
  File "/usr/lib/python3.7/site-packages/grub/GrubConf.py", line 440, in parse
    setattr(self, self.commands[com], arg_strip)
  File "/usr/lib/python3.7/site-packages/grub/GrubConf.py", line 238,
in _set_default
    if self._default < 0:
TypeError: '<' not supported between instances of 'str' and 'int'


Hmmm?

So now try going back and try keeping things as "bytes"
for as long as possible


bash-5.0# vim /usr/lib/python3.7/site-packages/grub/GrubConf.py +376

bash-5.0# diff  GrubConf.py /usr/lib/python3.7/site-packages/grub/GrubConf.py
376c376
<             lines = buf.split("\n")
---
>             lines = buf.split(b"\n")

bash-5.0# /usr/lib/xen/bin/pygrub --debug --offset=1048576
--list-entries /dev/vg_xen_vbds/lv_4g_02
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
Traceback (most recent call last):
  File "/usr/lib/xen/bin/pygrub", line 907, in <module>
    chosencfg = run_grub(file, entry, fs, incfg["args"])
  File "/usr/lib/xen/bin/pygrub", line 625, in run_grub
    g = Grub(file, fs)
  File "/usr/lib/xen/bin/pygrub", line 249, in __init__
    self.read_config(file, fs)
  File "/usr/lib/xen/bin/pygrub", line 460, in read_config
    self.cf.parse(buf)
  File "/usr/lib/python3.7/site-packages/grub/GrubConf.py", line 388, in parse
    if l.startswith('#'):
TypeError: startswith first arg must be bytes or a tuple of bytes, not str


bash-5.0# vim /usr/lib/python3.7/site-packages/grub/GrubConf.py +388

bash-5.0# diff  GrubConf.py /usr/lib/python3.7/site-packages/grub/GrubConf.py
376c376
<             lines = buf.split("\n")
---
>             lines = buf.split(b"\n")
388c388
<             if l.startswith('#'):
---
>             if l.startswith(b'#'):


bash-5.0# /usr/lib/xen/bin/pygrub --debug --offset=1048576
--list-entries /dev/vg_xen_vbds/lv_4g_02
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
Traceback (most recent call last):
  File "/usr/lib/xen/bin/pygrub", line 907, in <module>
    chosencfg = run_grub(file, entry, fs, incfg["args"])
  File "/usr/lib/xen/bin/pygrub", line 625, in run_grub
    g = Grub(file, fs)
  File "/usr/lib/xen/bin/pygrub", line 249, in __init__
    self.read_config(file, fs)
  File "/usr/lib/xen/bin/pygrub", line 460, in read_config
    self.cf.parse(buf)
  File "/usr/lib/python3.7/site-packages/grub/GrubConf.py", line 392, in parse
    if l.startswith('function'):
TypeError: startswith first arg must be bytes or a tuple of bytes, not str


bash-5.0# vim /usr/lib/python3.7/site-packages/grub/GrubConf.py +392


bash-5.0# diff  GrubConf.py /usr/lib/python3.7/site-packages/grub/GrubConf.py
376c376
<             lines = buf.split("\n")
---
>             lines = buf.split(b"\n")
388c388
<             if l.startswith('#'):
---
>             if l.startswith(b'#'):
392c392
<             if l.startswith('function'):
---
>             if l.startswith(b'function'):

bash-5.0# /usr/lib/xen/bin/pygrub --debug --offset=1048576
--list-entries /dev/vg_xen_vbds/lv_4g_02
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
Traceback (most recent call last):
  File "/usr/lib/xen/bin/pygrub", line 907, in <module>
    chosencfg = run_grub(file, entry, fs, incfg["args"])
  File "/usr/lib/xen/bin/pygrub", line 625, in run_grub
    g = Grub(file, fs)
  File "/usr/lib/xen/bin/pygrub", line 249, in __init__
    self.read_config(file, fs)
  File "/usr/lib/xen/bin/pygrub", line 460, in read_config
    self.cf.parse(buf)
  File "/usr/lib/python3.7/site-packages/grub/GrubConf.py", line 401, in parse
    title_match = re.match('^menuentry ["\'](.*?)["\'] (.*){', l)
  File "/usr/lib/python3.7/re.py", line 173, in match
    return _compile(pattern, flags).match(string)
TypeError: cannot use a string pattern on a bytes-like object


Hmmm?

This looks like something a bit harder to transpose into
a "bytes" paradigm.

Any ideas as to why my pythin3 Grub works in some cases but not
in others?

Kevin Buckley.

_______________________________________________
Grub-devel mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/grub-devel