[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[PATCH 0/3] Add support for EFI file system transposition
From: |
Pete Batard |
Subject: |
[PATCH 0/3] Add support for EFI file system transposition |
Date: |
Mon, 6 Jun 2022 17:50:30 +0100 |
Hello everyone,
This series of patches adds file system transposition support, for UEFI boot
media created with grub-mkrescue.
File system transposition means the ability to take the content of a UEFI
bootable media and copy it, at the file system level, to a partition that was
independently created and formatted by the user, while preserving the ability
of the media to boot in UEFI mode.
We see this as a much needed improvement to GRUB when one of the core concept
of EFI is to do away with the requirement to have to create boot media at the
block level, one of the major pain points of BIOS systems' users.
Currently, grub-mkrescue fails to meet the goal of UEFI file system
transposition on 3 accounts:
1. It does not include file system support for FAT or NTFS, whereas these are
the native file systems supported by UEFI (with FAT being mandatory per UEFI
specs, and with NTFS being found more and more commonly on x64 commodity
hardware such as, from my direct experience, about any motherboards that has
been produced by ASUS, Gigabyte or Intel in the past 10 years).
2. It uses a efi.img to embed the UEFI bootloaders, but does not keep a copy of
these bootloaders on the ISO9660 file system itself, with the end result that,
when copying the media at the file system level, the '/efi/boot/' directory and
its content is missing.
3. It relies on volume UUID to locate the boot media, a method that does not
survive transposition when the content is copied to a newly user-created
partition.
The following patches fix each one of these issues. More specifically:
1. Adding fat and ntfs support can easily be added as additional modules and,
considering that these are file systems natively supported on commonplace UEFI
hardware, the benefits vastly outweighs the very limited increase in size.
2. Duplicating the 'efi.img' bootloaders onto the ISO9660 file system is also
easily accomplished by dropping the use of a temporary directory to generate
the 'efi.img' and instead moving copying that content to the ISO9660 root
level. At this stage, we will point out that we consider it should really be
the job of xorriso, rather than grub-mkrescue, to accomplish this duplication
(hence why I am CC'ing Thomas), but we don't know the technical difficulties
that result from trying to map back the content of a FAT image back onto the
ISO9660 structure. There again, in terms of increase in size, we see the
cost/benefit ratio as non issue.
3. Searching for the boot media is now be carried by looking for a
'/.disk/<TIMEBASED_UUID>.uuid' file rather than an actual partition UUID, as
real world usage does show that relying on specific labels or UUIDs being
assigned to specific partitions is actually a brittle solution. Note that we
only carry out these alterations for EFI boot, and don't modify the existing
BIOS/Legacy media search method. It should also be noted that the reason we
chose a '.disk/' directory to place the UUID file is because '.disk/' has
become a de-facto standard to place disk related content for Debian and Ubuntu,
which, in the fragmented world of Linux distribution, is as good as a standard
as you can get. We did briefly consider using '/System/', but decided against
it as this latter directory is geared towards MacOS usage, and we see going
with a more generic dot directory as a better approach. We have validated that,
if the added content already contains a '.disk/' directory, then our new
'.disk/#####.uuid' file does get properly merged with that content.
With these limited changes, grub-mkrescue can now be used to produce media that
properly survives file system transposition, thereby satisfying one of the
implicit goals of EFI of allowing end-users to carry out the creation of
bootable EFI media at the file system level exclusively.
Now, because this is relevant to this patchset (it's pretty much the motivation
behind it), and because it is my understanding that some of the people creating
ISOHybrid boot media, especially if they are using UNIX derivatives as their
main work environments, are not always aware of the issues that arise from
treating ISOHybrid as a glorified dd image, I will use this opportunity to
elaborate on issues that arise from dealing with ISOHybrid media that lacks
support for file system transposition.
But first I should point out that these notes should not be construed as
criticism of ISOHybrid per se, as we very much recognise the great benefits of
ISOHybrid and the major technical achievements that went into it. We however do
see it as important to try to debunk the idea that, once you have an ISOHybrid
image that works well for both dd and optical, your job is done and remind
people who place their trust in ISOHybrid image that, for the reasons
highlighted below, having a working dd image is only half the job.
And so, without further ado:
* One massive real-world problem, that Linux-oriented people tend to ignore, is
that for the most widely used Operating System out there (Windows) the
"ISOHybrid = dd" approach is a *MASSIVE* pain point for users, on account that
the resulting media is either something that Windows will not mount, or only
mount the ESP of, with the direct result that a fait amount of users assume
that their media was not properly created and, therefore, don't even try to
boot it. I genuinely cannot stress enough how this is far from being a limited
problem, or something that one should feel entitled to ask Windows users to
"just plow through". See for instance [1], [2], [3], [4], [5], [6], [7], [8],
[9], [10], [11], [12] if you need some convincing of this, knowing that I could
easily add a dozen more refs, and that I didn't have to look further than a
couple days ago to find a recent one [4]. Thus, one of the results of
*restricting* the creation of ISOHybrid media on Windows, through the use of
block copy only, is that this will result in many users having a very poor
first time experience of your ISOHybrid-contained software...
* Having a file system transposable UEFI media does open the door to some very
desirable usage, such as single media OS installation (See for instance the
installation of vanilla Debian on Pi 4 using a single media as installer and
target [13]), which can be critical for of users of SoC based systems that do
not have the luxury or resources to run to their (not-always-so-local) corner
shop in order to buy another flash drive. Currently, I don't really see
dd-written ISOHybrid media being able to provide that kind of feature whereas
it is easy envision all kind of other scenarios where file system transposition
can be a boon for users, such as allowing the copying of non-free WiFi firmware
blobs onto the boot media, in order to be able to proceed with a networked
installation...
* Formatting a partition, and even repartitioning a drive, is typically a lot
less error prone for non tech-savvy users than using a dd like utility, in
terms of preventing the erasure of the wrong disk. This is especially true on
non Linux platforms where GUI tools or context menus will usually guide the
user into selecting the appropriate target.
* dd writing a GPT-based ISOHybrid media does result in a disk that has an
invalid backup GPT, on account that one never has a media that is the exact
same size as the original image, thereby resulting in garbage residing in the
last 33 sectors of the disk. Some pedantic UEFI firmwares, as well as some
ill-programmed OSes (e.g. Windows 7 will BSOD on this) may take objection to an
improper backup GPT...
* As mentioned earlier, one of the established goals of EFI was to get rid of
the requirement to write any parts of a media at the block level to make it
bootable. As such, it has been quite surprising to see a lot of the proponents
of ISOHybrid to push for a complete 180 on this goal, by enforcing the use of
block level tools. This is even more problematic as, for Windows users, this
means installation of third-party software, which some users may rightfully
take objection to, in terms of not adding potential holes into a trust chain
that is critical for OS installation.
* Finally, there's really no reason why GRUB (and by extension Linux) boot
media should remain inferior to Windows boot media in that respect, since
Windows images have been designed to support EFI file system transposition for
years. Thankfully, some major Linux distributions, such as Debian, Ubuntu and
(for the most part) Arch, have grasped the importance of file system
transposition for their ISOHybrid media, and are making a concerted effort to
support it. But we do feel that it should very much have been the job of GRUB
to lead the way in that respect.
Regards,
/Pete
[1]
https://old.reddit.com/r/linuxquestions/comments/j4nolf/creating_a_bootable_usb_drive_for_linux
[2]
https://old.reddit.com/r/ManjaroLinux/comments/gofq71/problem_with_rufus_310_and_manjaro_2001
[3]
https://old.reddit.com/r/ManjaroLinux/comments/gjdpi4/cannot_create_bootable_usb_usb_size_shrinks_after
[4]
https://old.reddit.com/r/Proxmox/comments/v2en1r/proxmox_couldnt_find_iso_before_launching_setup/iarz6hb/?context=3
[5]
https://old.reddit.com/r/techsupport/comments/499b5c/usb_stick_capacity_shrunk_to_2mb
[6]
https://superuser.com/questions/752874/16-gb-usb-flash-drive-capacity-down-to-938-mb
[7]
https://forums.tomshardware.com/threads/usb-flash-drive-8gb-is-now-only-1gb.660997/
[8] https://www.eassos.com/blog/how-to-restore-usb-drive-back-to-full-capacity
[9] https://www.easeus.com/partition-master/fix-usb-drive-incorrect-size.html
[10]
https://www.quora.com/After-making-my-32-GB-pen-drive-Kali-Linux-bootable-its-size-reduced-to-712-KB-Is-it-normal-If-not-then-how-can-I-fix-it
[11]
https://askubuntu.com/questions/289971/usbs-storage-capacity-reduced-to-2-mb-from-16-gb
[12]
https://askubuntu.com/questions/611325/capacity-of-pen-drive-shown-is-less-than-the-actual
[13] https://forums.raspberrypi.com/viewtopic.php?f=50&t=282839
Pete Batard (3):
grub-mkrescue: Add support for FAT and NTFS on EFI boot
grub-mkrescue: Preserve a copy of the EFI bootloaders on the ISO9660
file system
grub-mkrescue: Search by file UUID file rather than partition UUID for
EFI boot
util/grub-mkrescue.c | 60 ++++++++++++++++++++++++++++++++++----------
1 file changed, 47 insertions(+), 13 deletions(-)
--
2.36.0.windows.1
- [PATCH 0/3] Add support for EFI file system transposition,
Pete Batard <=
- [PATCH 2/3] grub-mkrescue: Preserve a copy of the EFI bootloaders on the ISO9660 file system, Pete Batard, 2022/06/06
- [PATCH 1/3] grub-mkrescue: Add support for FAT and NTFS on EFI boot, Pete Batard, 2022/06/06
- [PATCH 3/3] grub-mkrescue: Search by file UUID file rather than partition UUID for EFI boot, Pete Batard, 2022/06/06
- Re: [PATCH 0/3] Add support for EFI file system transposition, Thomas Schmitt, 2022/06/07
- Re: [PATCH 0/3] Add support for EFI file system transposition, Michael Schierl, 2022/06/12