Re: [Libcdio-devel] [PATCH v2 0/4] Add El Torito virtual image support

[Pete Batard]

Hi all,

So... we have a problem:

On 2024.01.29 15:04, Thomas Schmitt via Libcdio-devel wrote:
> Pete Batard wrote:
> > I'll just add that I didn't formally test this, since I haven't had a chance
> > to locate an ISO with a > 32MB El-Torito image yet.
>
> You can build yourself one:
>
> - Get legacy PC-BIOS boot image isolinux.bin from e.g. a Debian netinst
>    ISO:
>
>      xorriso -osirrox on -indev debian-12.2.0-amd64-netinst.iso \
>              -extract /isolinux/isolinux.bin isolinux.bin
>
>    Or just make a fake file:
>
>      dd if=/dev/zero bs=1K count=40 of=isolinux.bin
>
>
> - Make yourself a FAT filesystem image >= 32 MiB or just some large dummy
>    file named efi.img :
>
>      dd if=/dev/zero bs=1M count=40 of=efi.img
>
>
> - Pack up as bootable ISO for legacy and for EFI:
>
>      xorriso -as mkisofs -v -o test.iso \
>              -c /boot.cat \
>              -b /isolinux.bin \
>                 -no-emul-boot -boot-info-table -boot-load-size 4 \
>              -eltorito-alt-boot
>              -e /efi.img -no-emul-boot \
>              ./isolinux.bin ./efi.img
>
>    With contemporary xorriso expect messages like
>
>    xorriso : WARNING : Boot image load size exceeds 65535 blocks of 512 bytes. 
> Will record 0 in El Torito to extend ESP to end-of-medium.
>    xorriso : WARNING : EFI boot equipment is provided but no directory /EFI/BOOT
>    xorriso : WARNING : will emerge in the ISO filesystem. A popular method to
>    xorriso : WARNING : prepare a USB stick on MS-Windows relies on having in the
>    xorriso : WARNING : ISO filesystem a copy of the EFI System Partition tree.
>    ISO image produced: 20683 sectors
>
>    If you do not see the warning about "size exceeds 65535 blocks" then
>    your xorriso might be too old for the purpose.
>    (https://www.gnu.org/software/xorriso/xorriso-1.5.6.pl02.tar.gz
>     is the current version, easy to compile, and works without system-wide
>     installation.)

Thanks for this.

I followed these instructions and was able to build an ISO with a > 32 
MB El Torito image. In case this matters, I've been using Debian's 
default xorriso v1.5.4.

Testing this image with the new proposal (btw you can find an updated 
version of the proposal you can clone at 
https://github.com/pbatard/libcdio/tree/eltorito), I found that the 
reported size was 0 instead of the expected 40 MB.

Now, looking into this using iso-info (to the the LSNs for the images) I 
got:

------------------------------------------------------------------------
/[boot]/:
  - [LSN  20514]      2048 Jan 29 2024 20:34:45  0-boot-noemul.img
  - [LSN     34]         0 Jan 29 2024 20:34:45  1-boot-noemul.img
------------------------------------------------------------------------

With the "0-boot-noemul.img" having the expected size, but 
"1-boot-noemul.img" reporting 0 instead of 41943040.

Looking at the LSNs, we can quickly deduce the reason: xorriso happens 
to place that second image before the first image on the ISO, and our 
code expects that the image that gets expanded is the one with the 
highest LSN (there's a for loop that searches for it), so, obviously, 
the only El Torito image we might ever expand is the one at LSN 20514, 
but with its size set to 4 (virtual sectors = 1 ISO block), it doesn't 
need expansion.

Of course, the more interesting thing is that our logic of only 
expanding the image with the highest LSN complies with the UEFI specs 
that states the following (v2.10, section 13.3.2.1 ISO-9660 and El Torito):

> The CD-ROM format is completely described by the “El Torito” Bootable
> CD-ROM Format Specification Version 1.0. To boot from a CD-ROM or
> DVD-ROM in the boot services environment, an EFI System partition is
> stored in a “no emulation” mode as defined by the “El Torito”
> specification. A Platform ID of 0xEF indicates an EFI System
> Partition. The Platform ID is in either the Section Header Entry or
> the Validation Entry of the Booting Catalog as defined by the “El
> Torito” specification.
> EFI differs from “El Torito” “no emulation” mode in that it does not
> load the “no emulation” image into memory and jump to it. EFI
> interprets the “no emulation” image as an EFI system partition. EFI
> interprets the Sector Count in the Initial/Default Entry or the
> Section Header Entry to be the size of the EFI system partition. If
> the value of Sector Count is set to 0 or 1, EFI will assume the
> system partition consumes the space from the beginning of the “no
> emulation” image to the end of the CD-ROM.

The most important part from the above is the very last sentence. If you 
are going to set a EFI partition to a size of 0 or 1, then the image 
will occupy all sectors from the LSN to the end of the volume.

Therefore, on systems that comply with the UEFI specs, that 
1-boot-noemul.img image at sector 34 will actually include the data from 
the 0-boot-noemul.img at LSN 20514, and the size of the image that the 
system will process will be further extended by 2048 bytes... which 
makes me think that xorriso is somewhat abusing the UEFI specs.

Especially, as opposed to what Thomas stated in his other reply, nowhere 
does it states that: "The real size is the size of the FAT filesystem in 
such an image". Instead, UEFI is quite explicit that the real size is 
the size from the LSN to the end of the volume.

Obviously, adding an extra 2048 block of data to what is expected (but 
does not need to be -- one can have a UEFI firmware boot from whatever 
file system you want, and it's relatively easy to inject a non FAT 
module in a UEFI firmware on a system to have it handle other file 
systems) FAT is not expected to create much of an issue, at least for 
real-life public ISOs. But this unexpected abuse of the specs does 
create yet another issue that we need to address.

So I'm going to ask Thomas, though I suspect the answer is "ISOHybrid 
tweaking", is: Why is xorriso is placing the images that way?

I also have to state that modifying the current proposal to handle 
xorriso's way of placing the ESP is going to be problematic because for 
one thing, I don't see it as a sure thing that we can get the actual 
size from the image itself (again, even outside of running systems that 
support more than FAT natively, someone can very much create non FAT 
images, with 0xef platform ID, for testing or whatnot), so we're going 
to have to craft a new codepath for the non-compliant xorriso layout, 
and I'm not too happy with the straightforward/low-effort approach of 
treating every image with a size lower or equal to 1 as an expanded 
image (since I am very much trying to avoid having to sink time into 
full blown processing of platform id and whatnot).

And before someone replies that the current proposal also partially 
relies on the ability of a system to potentially read more than it 
really needs (in case of a 512-byte image that happens to reside on an 
ISO that contains more than 32 MB of data after it, with that data not 
being used for anything El Torito), I would say that there's a big 
difference between what one expects to be an unlikely scenario 
(probability of someone really wanting to use the smallest image size * 
probability of them crafting an ISO that has data after the El Torito 
section * probability that this data is more than 32 MB) that is being 
induced by the mastering application user, and what an mastering 
application happens to do with the default options (or at least what I 
expect to be very widely used ones when creating a BIOS+UEFI bootable ISO).

Finally, for the record, since I am also looking at what 7-zip does, it 
also happens that 7-zip has the same issue, i.e. it sees the 40 MB image 
as having a 0-byte size and cannot open or extract it.

Regards,

/Pete