bug#36747: Official MesCC bootstrap binaries differ from my locally built ones

[Mark H Weaver]

Hi Marius,

Marius Bakke <address@hidden> writes:

> Marius Bakke <address@hidden> writes:
>
>> Jan Nieuwenhuizen <address@hidden> writes:
>>
>>> Mark H Weaver writes:
>>>
>>> Hi Mark,
>>>
>>>>> I called that `wip-binaries', @master from three weeks ago.
>>>>
>>>> Thank you, that was a good start.  I found that some additional patches
>>>> were needed to match the bootstrap binaries that 'core-updates' is
>>>> currently based on.
>>>>
>>>> I ended up deleting and repushing a revised 'wip-binaries' to Savannah.
>>>> It includes slightly modified versions of the two commits you had
>>>> included, as well as some additional cherry-picked commits of yours to
>>>> update mescc-tools and add linux-libre-headers-bootstrap-tarball, and a
>>>> few of my own.
>>>
>>> Very nice.
>>>
>>>> I built the new bootstrap tarballs at the new 'wip-binaries', commit
>>>> c67becb31c30a5cd7685f166970ac4793e3a34a9, and here's what I got:
>>>>
>>>> mhw@jojen ~/guix-wip-binaries$ git describe
>>>> v1.0.1-2404-gc67becb31c
>>>> mhw@jojen ~/guix-wip-binaries$ ./pre-inst-env guix build 
>>>> --system=i686-linux bootstrap-tarballs
>>>> /gnu/store/bg086i2qw1fn2jgbd15d9v91hyjrjsb2-bootstrap-tarballs-0
>>>> mhw@jojen ~/guix-wip-binaries$ cd 
>>>> /gnu/store/bg086i2qw1fn2jgbd15d9v91hyjrjsb2-bootstrap-tarballs-0
>>>> mhw@jojen 
>>>> /gnu/store/bg086i2qw1fn2jgbd15d9v91hyjrjsb2-bootstrap-tarballs-0$ 
>>>> sha256sum *
>>>> 3e50c070a100b6bcf84c4bf5c868f9cd0a9fd1570f5d82fbfb78f8411959091b  
>>>> guile-static-stripped-2.2.4-i686-linux.tar.xz
>>>> 1acd8f83e27d2fac311a5ca78e9bf11a9a1638b82469870d5c854c4e7afaa26a  
>>>> linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz
>>>> 021543d9bb6af55f39e68d69692e3cb74646ced2cad0bb9ac0047ef81e9d7330  
>>>> mescc-tools-static-stripped-0.5.2-0.bb062b0-i686-linux.tar.xz
>>>> fb32090071b39fc804fb9a7fba96f0bc5eb844a0efd268fb24c42e6bfa959de0  
>>>> mes-minimal-stripped-0.19-i686-linux.tar.xz
>>>> c80cdd17b0a24eebdd75570ff72c4ec06e129bd702ac008186b57f6301c448e7  
>>>> static-binaries-0-i686-linux.tar.xz
>>>
>>>> Can you try "guix build --system=i686-linux bootstrap-tarballs" at the
>>>> new 'wip-binaries' branch and see if you get the same results?
>>>
>>> Yes, on c67becb31c30a5cd7685f166970ac4793e3a34a9 running
>>> "./pre-inst-env guix build --system=i686-linux bootstrap-tarballs" gives me 
>>> exactly this,
>>> also for guile-static-stripped! \o/
>>>
>>>> Also, I have a question: One of the changes I made to 'wip-binaries' was
>>>> to update mescc-tools to 0.5.2-0.bb062b0, to match the
>>>> %bootstrap-mescc-tools that's currently being used in 'core-updates'.
>>>>
>>>> However, I noticed that you have also apparently built the official
>>>> release of mescc-tools-0.5.2, which is on your site:
>>>>
>>>>   
>>>> http://lilypond.org/janneke/guix/20190722/mescc-tools-static-stripped-0.5.2-i686-linux.tar.xz
>>>>
>>>> and that this tarball is identical to the build output of the later git
>>>> commit: mescc-tools-static-stripped-0.5.2-0.bb062b0-i686-linux.tar.xz.
>>>>
>>>> With this in mind, could we just use 0.5.2?  What changed between 0.5.2
>>>> and 0.5.2-0.bb062b0, and what was the rationale for updating to bb062b0?
>>>
>>> Good catch.  We probably can, we might try that.
>>>
>>> I think the need for updating to bb062b0 has been removed during the
>>> review of the integration of the reduced binary seed bootstrap into
>>> core-updates by Ludovic.
>>>
>>> For historical reasons, I think this mescc-tools commit
>>>
>>> --8<---------------cut here---------------start------------->8---
>>> commit c184e95096881a13f29ebd7fc507fe305d3d8de5 (gitlab/janneke, janneke)
>>> Author: Jan Nieuwenhuizen <address@hidden>
>>> Date:   Thu Oct 4 22:03:31 2018 +0200
>>>
>>>     build.sh: Update for mes 0.18.
>>> --8<---------------cut here---------------end--------------->8---
>>>
>>> was needed at a time that we did not have mescc-tools or mes in
>>> bootstrap tarballs.  We built bootstrap variants of mescc-tools and mes
>>> using a externally (outside fo Guix) built mescc-tools-seed and
>>> (an almost pure ASCII) mes-seed.
>>
>> I tried building the i686 bootstrap tarballs from wip-binaries with this
>> additional patch:
>>
>> diff --git a/gnu/packages/mes.scm b/gnu/packages/mes.scm
>> index e298cb05c1..380cac6c88 100644
>> --- a/gnu/packages/mes.scm
>> +++ b/gnu/packages/mes.scm
>> @@ -139,33 +139,31 @@ Guile.")
>>      (license gpl3+)))
>>  
>>  (define-public mescc-tools
>> -  (let ((commit "bb062b0da7bf2724ca40f9002b121579898d4ef7")
>> -        (revision "0")
>> -        (version "0.5.2"))
>> -    (package
>> -      (name "mescc-tools")
>> -      (version (string-append version "-" revision "." (string-take commit 
>> 7)))
>> -      (source (origin
>> -                (method url-fetch)
>> -                (uri (string-append
>> -                      
>> "https://git.savannah.nongnu.org/cgit/mescc-tools.git/snapshot/";
>> -                      name "-" commit
>> -                      ".tar.gz"))
>> -                (sha256
>> -                 (base32
>> -                  "1h6j57wyf91i42b26f8msbv6451cw3nm4nmpl1fckp9c7vi8mwkh"))))
>> -      (build-system gnu-build-system)
>> -      (supported-systems '("i686-linux" "x86_64-linux"))
>> -      (arguments
>> -       `(#:make-flags (list (string-append "PREFIX=" (assoc-ref %outputs 
>> "out")))
>> -         #:test-target "test"
>> -         #:phases (modify-phases %standard-phases
>> -                    (delete 'configure))))
>> -      (synopsis "Tools for the full source bootstrapping process")
>> -      (description
>> -       "Mescc-tools is a collection of tools for use in a full source
>> +  (package
>> +    (name "mescc-tools")
>> +    (version "0.5.2")
>> +    (source (origin
>> +              (method url-fetch)
>> +              (uri (string-append
>> +                    
>> "https://git.savannah.nongnu.org/cgit/mescc-tools.git/snapshot/";
>> +                    name "-Release_" version
>> +                    ".tar.gz"))
>> +              (file-name (string-append name "-" version ".tar.gz"))
>> +              (sha256
>> +               (base32
>> +                "01x7bhmgwyf6mc2g1hcvibhps98nllacqm4f0j5l51b1mbi18pc2"))))
>> +    (build-system gnu-build-system)
>> +    (supported-systems '("i686-linux" "x86_64-linux"))
>> +    (arguments
>> +     `(#:make-flags (list (string-append "PREFIX=" (assoc-ref %outputs 
>> "out")))
>> +       #:test-target "test"
>> +       #:phases (modify-phases %standard-phases
>> +                  (delete 'configure))))
>> +    (synopsis "Tools for the full source bootstrapping process")
>> +    (description
>> +     "Mescc-tools is a collection of tools for use in a full source
>>  bootstrapping process.  It consists of the M1 macro assembler, the hex2
>>  linker, the blood-elf symbol table generator, the kaem shell, exec_enable 
>> and
>>  get_machine.")
>>      (home-page "https://savannah.nongnu.org/projects/mescc-tools";)
>> -    (license gpl3+))))
>> +    (license gpl3+)))

I guess this is equivalent to reverting commit
78ced7975b0665e810834391d826c9f0ef7277e1 on the 'wip-binaries' branch,
yes?

>> And got this result:
>>
>> $ cd /gnu/store/rdwyr8mh7dvhfkb5g4cws6q40hp23rbi-bootstrap-tarballs-0
>> $ sha256sum *
>> 3e50c070a100b6bcf84c4bf5c868f9cd0a9fd1570f5d82fbfb78f8411959091b  
>> guile-static-stripped-2.2.4-i686-linux.tar.xz
>> 1acd8f83e27d2fac311a5ca78e9bf11a9a1638b82469870d5c854c4e7afaa26a  
>> linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz
>> 021543d9bb6af55f39e68d69692e3cb74646ced2cad0bb9ac0047ef81e9d7330  
>> mescc-tools-static-stripped-0.5.2-i686-linux.tar.xz
>> fb32090071b39fc804fb9a7fba96f0bc5eb844a0efd268fb24c42e6bfa959de0  
>> mes-minimal-stripped-0.19-i686-linux.tar.xz
>> c80cdd17b0a24eebdd75570ff72c4ec06e129bd702ac008186b57f6301c448e7  
>> static-binaries-0-i686-linux.tar.xz

Great, looks good to me!

>> I also merged the branch to core-updates and reverted the bash patch,

I think that 'bash-4.4-linux-pgrp-pipe.patch', adapted to 5.0, should be
applied to all 'bash' packages in 'core-updates'.  To be more precise,
the patch should be applied to the main 'bash' package in bash.scm,
inherited from all other bash packages, so that PGRP_PIPE is set
unconditionally set on systems based on Linux (the kernel), regardless
of the kernel version running on the build machine.

>> which produced this derivation for "guix build -d -s i686-linux
>> bootstrap-tarballs":
>>
>> /gnu/store/ld82vli1msfrlimjaryznrqcwm0jc5ii-bootstrap-tarballs-0.drv
>
> Here are the hashes from this derivation:
>
> $ cd /gnu/store/srsqilb3g70r8c7vma0gpam21z1zmg2w-bootstrap-tarballs-0
> $ sha256sum *
> 4e5b219be4d9ad4d125f17b8a8a991e78be3908aadc8d22d1a115e96ec856279  
> guile-static-stripped-2.2.6-i686-linux.tar.xz
> 9e34b26526b184912b0d97df96e316151fd97ede24aec2f22dd13ed33438b2e4  
> linux-libre-headers-stripped-4.19.56-i686-linux.tar.xz
> 7db07a7097a7920e17f6f1794098b9f29bc3522c36a33cf797123f64581cdc34  
> mescc-tools-static-0.5.2-i686-linux.tar.xz
> 3a9b050c1a2b61bb1cbed16ee07aecdc83e9097ce4a914623aa1b9808abe57f4  
> mes-minimal-stripped-0.19-i686-linux.tar.xz
> 1ccb4f39656eb977706db7a4aa811461785200b0a11e847035bb89fef91b85ab  
> static-binaries-0-i686-linux.tar.xz

What was the purpose of building these?  'core-updates' is built upon on
the earlier, unverified, reduced binary seed bootstrap binaries.

> @Mark: do you think we are ready to merge this now?  Can you do it?

I think what needs to be done is the following:

(1) commit 78ced7975b0665e810834391d826c9f0ef7277e1 on 'wip-binaries'
    should be reverted, to downgrade mescc-tools to the 0.5.2 release.

(2) The 'wip-binaries' tarballs should be uploaded to a new subdirectory
    of <https://alpha.gnu.org/gnu/guix/bootstrap/i686-linux/>, along
    with digital signatures, of course.  I'm talking about these in
    particular:

    3e50c070a100b6bcf84c4bf5c868f9cd0a9fd1570f5d82fbfb78f8411959091b  
guile-static-stripped-2.2.4-i686-linux.tar.xz
    1acd8f83e27d2fac311a5ca78e9bf11a9a1638b82469870d5c854c4e7afaa26a  
linux-libre-headers-stripped-4.14.67-i686-linux.tar.xz
    021543d9bb6af55f39e68d69692e3cb74646ced2cad0bb9ac0047ef81e9d7330  
mescc-tools-static-stripped-0.5.2-i686-linux.tar.xz
    fb32090071b39fc804fb9a7fba96f0bc5eb844a0efd268fb24c42e6bfa959de0  
mes-minimal-stripped-0.19-i686-linux.tar.xz
    c80cdd17b0a24eebdd75570ff72c4ec06e129bd702ac008186b57f6301c448e7  
static-binaries-0-i686-linux.tar.xz

(3) The following bootstrap packages in 'core-updates' bootstrap.scm
    should be updated to use the new binaries above:

     (a) %bootstrap-linux-libre-headers
     (b) %bootstrap-mescc-tools
     (c) %bootstrap-mes

(4) Berlin should start rebuilding 'core-updates'.

If desired, steps (3) and (4) could come before (2) if someone
temporarily uploads the new binaries somewhere else, and adjusts
'%bootstrap-base-urls' accordingly.  The key is for the hashes and file
names to match what we've agreed on here, as I listed in (2) above.

What do you think?

It will be a couple of days before I can do more solid work on this, but
I'd be grateful if others could carry it forward in the meantime.

     Thanks!
       Mark