Re: GRUB2 Behavior Question

[Matt Sickler]

Thanks for the info John.

Here's some more background into our setup.  It's a semi-embedded system (ie, 
no "system admin" - all admin has to be handled by our programs/scripts) that 
is installed on hundreds/thousands of devices around the world (so manually 
doing things is possible but very expensive).

For the most part, we're using stock Debian 8 (Jessie).  There's a slight 
modification to the grub scripts in /etc/grub.d.   The change is I added 
sourcing my own script just after it sources the "grub-mkconfig_lib" library.
The contents of the new script is:
make_system_path_relative_to_its_root ()
{
    "${grub_mkrelpath}" "$1" | sed -e 's,/@[^/]*,/@,'
}

I wrote this quite a few years ago, so I don't remember *how* it works, but I 
remember it was required for a reason that I'll get to shortly.

Since we're using btrfs for the filesystem, we have it setup where the 
linux-root (I'll use this to refer to what linux mounts at /) is actually a 
subvolume.
The root of the btrfs filesystem has these things in it:
@                A symlink to the subvolume we want to boot into
boot            A symlink to "/@/boot"
@deb8       One of our linux-root subvolumes
@deb10     Another of our linux-root subvolumes (Really, the linux-root 
subvolumes could be named anything.)

In all the linux-root's, /etc/fstab has this:   UUID=...  /  btrfs   
defaults,subvol=@        0 1
The patch to grub that I mentioned at the top of the email is to prevent grub 
from "resolving" the @ symlink to the real subvolme path when generating the 
/boot/grub/grub.cfg file.
So our generated grub.cfg would end up with entries like this:  linux   
/@/boot/vmlinuz-3.16.0-9-amd64 root=UUID=06307c88-ee37-4a15-ada7-83bf6d8c2955 
ro single rootflags=subvol=@
(Without my patch, the path would be /@deb10/boot/vmlinuz-... instead.)

Now here's where the problem starts.  In the @deb8 subvolume, linux-3.16... is 
installed and in the @deb10 subvolume, linux-4.19... is installed.  The 
/@deb8/boot/grub/grub.cfg and /@deb10/boot/grub/grub.cfg files are both correct.
However, if grub-install is run while booted into the @deb10 subvolume, if we 
"switch" back to @deb8 (by changing that @ symlink and rebooting) when grub 
starts up it uses the config from /@deb10/boot/grub/grub.cfg instead of the one 
we'd expect (from /@deb8/boot/grub/grub.cfg<mailto:/@deb8/boot/grub/grub.cfg>).

I'm not at all up to speed how grub actually executes at boot, but I'm aware 
there are executable bits "outside" of the filesystem (in the MBR or 
somewhere?).   My theory is those pieces hold an inode number of 
/boot/grub/grub.cfg when grub-install is ran (so that it doesn't need to have 
lookup code to find the data given the filepath "/boot/grub/grub.cfg").   Or 
maybe some other kind of config caching?
We've found that if we run "update-grub" and "grub-install /dev/sda" whenever 
we "switch" snapshots, the problem doesn't occur.