Re: [Bug-xorriso] generating reproducible ISOs with xorriso

[Daniel Kahn Gillmor]

On Thu 2015-06-04 14:08:36 -0400, Thomas Schmitt wrote:
> The syntax would have to be different and probably a more
> comprehensive name will come to us when we know what xorriso
> features in particular shall be bundled with the new command.

That seems like a reasonable approach.

> The users will have their own problems to get the new concept
> integrated into their ISO production.
>
> The main users within Debian are debian-cd and Debian Live.
> One should develop proposals how to make their ISOs reproducible
> and how to publish the parameters (e.g. the time stamp) which
> they will have to provide to xorriso when reproducing the ISOs.
>
> A special user is grub-mkrescue. We will have to come up with
> a well motivated use case to convince Vladimir Serbinenko.

That's fine; we can also patch grub within debian to do the right thing
until we've gotten Vladimir properly convinced :) I've already convinced
Vladimir that he should use a stable embedding timestamp when creating
PE executables, so he's aware of the general concern about
reproducibility:

 
http://git.savannah.gnu.org/cgit/grub.git/commit/?id=85a7be2414c4718e96d81a2ebaa70d0d42152e62

I can imagine that he would also like to be able to say that
"grub-mkrescue", executed twice with the same arguments on different
machines (running the same versions of grub) should be able to produce
the same .iso image.

I'll think about how to pose that to the grub mailing list once we've
sorted out how to do it.

>> https://reproducible.debian.net/dbd/unstable/amd64/grub2_2.02~beta2-23.debbindiff.html
>> if i'm reading that right, then in
>> /usr/lib/grub-rescue/grub-rescue-cdrom.iso, we have
>> /BOOT/GRUB/GRUB.CFG;1 with extent 2316 in the first build, and extent 47
>> in the second.
>
> Strange ... i had a different model in mind.
> Will have to review the source of libisofs (ecma119.c, i guess).
>
> How were the two compared ISOs generated ?
> By the same xorriso version and same options resp. commands ?

yes, the same set of packages (the "build-deps", in debian), running
with the same source code (the grub source, which invokes xorriso).

You can see the full build output (warning! > 30MiB!) here:

 
https://reproducible.debian.net/rbuild/unstable/amd64/grub2_2.02~beta2-23.rbuild.log


It looks like it's being called from within grub-mkrescue, once for the
floppy and once for the CD image.  This comes from

   https://sources.debian.net/src/grub2/2.02~beta2-22/debian/rules/#L242

--------------------------------------
pkgdatadir=/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc \
        obj/grub-pc/grub-mkrescue \
        --directory=/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-core \
        
--locale-directory=/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-core/po 
\
        
--output=/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-rescue-cdrom.iso \
        /tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/rescue-disk
obj/grub-pc/grub-mkrescue: warning: cannot open directory 
`/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-core/po': No such file or 
directory.
xorriso 1.3.2 : RockRidge filesystem manipulator, libburnia project.

Drive current: -outdev 
'stdio:/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-rescue-cdrom.iso'
Media current: stdio file, overwriteable
Media status : is blank
Media summary: 0 sessions, 0 data blocks, 0 data, 83.4g free
Added to ISO image: directory '/'='/tmp/grub.MBEdSo'
xorriso : UPDATE : 277 files added in 1 seconds
Added to ISO image: directory 
'/'='/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/rescue-disk'
xorriso : UPDATE : 280 files added in 1 seconds
xorriso : NOTE : Copying to System Area: 512 bytes from file 
'/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-core/boot_hybrid.img'
ISO image produced: 2467 sectors
Written to medium : 2467 sectors at LBA 0
Writing to 
'stdio:/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-rescue-cdrom.iso' 
completed successfully.

# save space for floppy image
rm -rf obj/grub-rescue-pc/grub-core/po obj/grub-rescue-pc/unicode.pf2
pkgdatadir=/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc \
        obj/grub-pc/grub-mkrescue \
        --directory=/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-core \
        
--locale-directory=/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-core/po 
\
        
--output=/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-rescue-floppy.img 
\
        --compress=xz \
        -- -no-pad /tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/rescue-disk
obj/grub-pc/grub-mkrescue: warning: cannot open directory 
`/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-core/po': No such file or 
directory.
xorriso 1.3.2 : RockRidge filesystem manipulator, libburnia project.

Drive current: -outdev 
'stdio:/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-rescue-floppy.img'
Media current: stdio file, overwriteable
Media status : is blank
Media summary: 0 sessions, 0 data blocks, 0 data, 83.4g free
Added to ISO image: directory '/'='/tmp/grub.cFfL5C'
xorriso : UPDATE : 276 files added in 1 seconds
Added to ISO image: directory 
'/'='/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/rescue-disk'
xorriso : UPDATE : 279 files added in 1 seconds
xorriso : NOTE : Copying to System Area: 512 bytes from file 
'/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-core/boot_hybrid.img'
ISO image produced: 596 sectors
Written to medium : 596 sectors at LBA 0
Writing to 
'stdio:/tmp/buildd/grub2-2.02~beta2/obj/grub-rescue-pc/grub-rescue-floppy.img' 
completed successfully.
--------------------------------------

And it looks like grub-mkrescue can accept xorriso arguments.  from
grub-mkrescue(1):

       Generates a bootable CD/USB/floppy image.  Arguments other than options
       to this program are passed  to  xorriso,  and  indicate  source  files,
       source  directories, or any of the mkisofs options listed by the output
       of `xorriso -as mkisofs -help'.

       Option -- switches to native xorriso command mode.


So we could potentially just patch the invocation within debian/rules
initially, if there are flags that would be useful for tweaking the
extents...

> The sequence of extents can be influenced by the sort weight.
> (See man xorrisofs option --sort-weight, man xorriso -find action
>  sort_weight.)

I've read both of those sections but i'm at a loss as to what to supply
there to fiddle with the extents.  Any suggestions?

Regards,

      --dkg