Re: Menu for booting BTRFS snapshots

[Robert LeBlanc]

On Thu, Mar 2, 2017 at 7:40 PM, Jordan Uggla <address@hidden> wrote:
> Grub's view of things at boot is necessarily different from the view
> from your booted GNU/Linux environment, and indeed if you dual boot
> two GNU/Linux OSs they will each mount each other's volumes to
> different areas. Presumably /snaps/ is a mountpoint where you have
> configured that your btrfs be mounted, with the right mount options
> such that you can access your snapshots. Grub's grub.cfg is read
> before any OS's fstab, so from grub's point of view "/snaps/" is most
> likely just an empty directory. In addition, grub's view of a btrfs
> filesystem starts at the top level subvolume so the path to that
> "/snaps/" directory from grub is likely /@/snaps/ (though it depends
> on the naming scheme you and your distribution have used). You will
> most likely notice this when looking at other paths in your grub.cfg
> (from parts generated by grub-mkconfig). Thankfully grub has a handy
> utility for helping you find the path that grub will need to access a
> file at boot, given the path to a file on your booted system:
> grub-mkrelpath.
>
> TLDR: If you run "grub-mkrelpath /snaps/rxe-boot/grub2/grub.cfg" you
> will get as output the path that grub will need to access your
> grub.cfg at boot.
>
>
> Also, if you're just reading from that grubenv to get a list of
> subvolumes, you should know that grub can iterate over files and
> directories using for loops and globbing just like bash, and so
> (depending on your exact layout of snapshots) you can most likely
> dynamically generate a menu with one entry per snapshot. And by
> dynamically I don't mean "re-run grub-mkconfig to generate a new
> grub.cfg", I mean they can literally be detected at boot by grub using
> grub-script.

I understand what you are saying, but it doesn't match what I have
because I didn't provide enough information, so let me rectify that.
Here is my layout:

/
|-- root (subvolume)
|-- boot (subvolume)
|-- home (subvolume)
|-- snaps (directory)
|   |-- rxe-root (rw snap of /root)
|   |-- rxe-boot (rw snap of /boot)
|   |-- fresh-root (rw snap of /snaps-ro/fresh-root)
|   `-- fresh-boot (rw snap of /snaps-ro/fresh-boot)
`-- snaps-ro (directory)
    |-- fresh-root (ro snap of /root)
    `-- fresh-boot (ro snap of /boot)

/etc/fstab (root subvolume)
UUID=4b5cbe3a-57b8-444e-8f51-15e43c125a5d /
btrfs   subvol=root     0 0
UUID=4b5cbe3a-57b8-444e-8f51-15e43c125a5d /boot
btrfs   subvol=boot     0 0
UUID=4b5cbe3a-57b8-444e-8f51-15e43c125a5d /home
btrfs   subvol=home     0 0
UUID=a4e1eac9-930b-4000-9c88-4c54da9017b8 swap                    swap
   defaults        0 0
# Mount the complete btrfs in /mnt for ease.
UUID=4b5cbe3a-57b8-444e-8f51-15e43c125a5d /mnt/btrfs
btrfs   defaults        0 0

The fsab mounts the root subvolume at /, the boot subvolume at /boot,
the home subvolume at /home and the entire btrfs filesystem at
/mnt/btrfs. That way I can access all the snapshots easily and copy
using reflinks between any subvolumes. In each of the rw snapshots, I
modify the fstab to mount the appropriate subvolume for root and boot.

/etc/fstab (rxe-root snapshot)
UUID=4b5cbe3a-57b8-444e-8f51-15e43c125a5d /
btrfs   subvol=snaps/rxe-root     0 0
UUID=4b5cbe3a-57b8-444e-8f51-15e43c125a5d /boot
btrfs   subvol=snaps/rxe-boot     0 0
UUID=4b5cbe3a-57b8-444e-8f51-15e43c125a5d /home
btrfs   subvol=home     0 0
UUID=a4e1eac9-930b-4000-9c88-4c54da9017b8 swap                    swap
   defaults        0 0
# Mount the complete btrfs in /mnt for ease.
UUID=4b5cbe3a-57b8-444e-8f51-15e43c125a5d /mnt/btrfs
btrfs   defaults        0 0

Then in the rxe-boot snap I edit the /grub2/grub.cfg for one of the
menuentry items so that it loads the vmlinz and initramfs from the
snapshot directory and change the root value in the kernel command
line. I then reboot the box, load the config file from the rxe-boot
snap, boot into the edited menuentry. Once in the snapshot, I run
grub2-mkconfig to update all the menuentry items with the snapshot
path (it is smart enough to detect that it is now using a snapshot).
Here is an updated menuentry that I use to boot the snapshot.

menuentry 'CentOS Linux (4.9.13) 7 (Core)' --class centos --class
gnu-linux --class gnu --class os --unrestricted $menuentry_id_option
'gnulinux-4.9.13-advanced-4b5cbe3a-57b8-444e-8f
51-15e43c125a5d' {
       load_video
       insmod gzio
       insmod part_msdos
       insmod btrfs
       set root='hd0,msdos2'
       if [ x$feature_platform_search_hint = xy ]; then
         search --no-floppy --fs-uuid --set=root
--hint-bios=hd0,msdos2 --hint-efi=hd0,msdos2
--hint-baremetal=ahci0,msdos2 --hint='hd0,msdos2'
4b5cbe3a-57b8-444e-8f51-15e43c125a5d
       else
         search --no-floppy --fs-uuid --set=root
4b5cbe3a-57b8-444e-8f51-15e43c125a5d
       fi
       linux16 /snaps/rxe-boot/vmlinuz-4.9.13
root=UUID=4b5cbe3a-57b8-444e-8f51-15e43c125a5d ro
rootflags=subvol=snaps/rxe-root crashkernel=auto video=1024x768
panic=1 consoleblank=0
       initrd16 /snaps/rxe-boot/initramfs-4.9.13.img
}

When I boot the machine, to get to the snapshot kernels, I drop into a
GRUB command line shell and run

configfile /snaps/rxe-boot/grub2/grub.cfg

And it loads up the menu that is configured for the snapshot and I can
then select the edited menuentry and it boots up fine. I'm now trying
to automate this process to some degree.

So in the boot subvolume, I created a menuentry as specified in my
previous email named 'RXE Snapshot'.

menuentry 'RXE Snapshot' {
       load_env -f /snaps/rxe-boot/grub2/grubenv
       configfile /snaps/rxe-boot/grub2/grub.cfg
}

Then in the root subvolume, I run `grub2-set-default 'RXE Snapshot'`.
In the snapshot, the /boot/grub2/grubenv has a specific menuentry
specified. When I reboot the machine, 'RXE Snapshot' is correctly
selected as expected. Once the 5 second timeout happens, it loads the
config from the snapshot, but the correct menuentry is not selected
based on the grubenv from the snapshot.

/boot/grub2/grubenv (boot subvolume):
# cat /mnt/btrfs/boot/grub2/grubenv
# GRUB Environment Block
saved_entry=RXE Snapshot
######################################################################################################################################################################################
######################################################################################################################################################################################
######################################################################################################################################################################################
######################################################################################################################################################################################
######################################################################################################################################################################################
##############################################

/boot/grub2/grubenv (boot rxe-snapshot):
# cat /mnt/btrfs/snaps/rxe-boot/grub2/grubenv
# GRUB Environment Block
saved_entry=CentOS Linux (4.9.13) 7 (Core)
######################################################################################################################################################################################
######################################################################################################################################################################################
######################################################################################################################################################################################
######################################################################################################################################################################################
######################################################################################################################################################################################
##############################################

What I'm trying to accomplish is that when I load the configfile from
a snapshot, that it will also select the right kernel based on the
grubenv in _that_ snapshot. That way when I need to work in a
snapshot, I set the boot subvolme to load the snapshot config using
it's grubenv (ie. RXE Snapshot) . Then when I'm in the snapshot, if I
need to boot into a new kernel in the snapshot to test something, I
run grub2-set-default in the snapshot and type reboot and it will
automatically load up the new kernel from the snapshot.

I hope to achieve:

boot -> boot subvolume grub menu -> 'RXE Snapshot' automatically
selected -> rxe-boot grub menu -> 'New kernel' automatically selected
-> new snapshot kernel booted.

Without needing any intervention from me. That way I don't have to
have an IPMI session open to the server to select the correct snapshot
kernel. Right now everything through 'rxe-boot grub menu' is working.
Now I just need to get the right kernel for the snapshot selected.
Running `load_env -f /snaps/rxe-boot/grub2/grubenv` in the 'RXE
Snapshot' menuentry is not enough.

I hope that helps clarify things.

----------------
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904  C70E E654 3BB2 FA62 B9F1