Re: Linux-libre 5.8 and beyond

[Alexandre Oliva]

Hello, Mark,

On Aug 12, 2020, Mark H Weaver <mhw@netris.org> wrote:

> Mark H Weaver <mhw@netris.org> wrote:
>>> the linux-libre project periodically deletes most of its older
>>> tarballs, even if there are no accidents.

> Jason Self <jason@bluehome.net> responded:
>> Just FYI that git://linux-libre.fsfla.org/releases.git was created
>> mainly to solve that problem. Versions are now pretty much permanent.

> That's helpful, thanks.  I didn't know about this.  Out of curiosity, is
> this git repository advertised anywhere?

Not much.  It was mentioned back in the announcements of 5.7-gnu and a
few subsequent ones on social media; in the 5.7-gnu news entry in the
Linux-libre web site, and in the documentation we wrote for Guix
developers, that was sent to some of you not long ago.

Though it was announced sort of widely, since this move was directed
primarily at satisfying a Guix pain point, I figured I'd add it to
downloads only after making sure it did address Guix's needs, so that,
should it require significant changes, there wouldn't have to be much
concern about backward compatibility with the current status quo.

> One question: Would it solve the problem that I mentioned in my earlier
> email, namely the problem of how to determine which precise commit
> introduced a regression between two stable kernel releases?

No.  There are much better (faster and less risky) ways to tend to that
requirement, see #bisecting below.


>>> It may be useful for users with newer hardware devices, which are
>>> not yet well supported by the latest stable release, to use an
>>> arbitrary commit from either Linus' mainline git repository or some
>>> other subsystem tree.
>> 
>> The cleaning up scripts are version-specific and won't work on an
>> "arbitrary commit from Linus's mainline git repository" (i.e., someone
>> wanting to get today's most recent commit going into 5.9.) The scripts
>> would fall over and die in such a scenario,

> Okay, perhaps this was wishful thinking on my part.

Yup.  If you ran a deblob-check in verify mode on the resulting
tarballs, you'd see how error-prone this is.  You'd at least stop
non-Free code from silently sneaking in and finding its way into running
on users' machines.  That's the *least* someone who runs the
deblob-scripts on their own should do to smoke-test the result WRT
*known* freedom issues.

> I had hoped that the deblob scripts would typically mostly work, even
> if they weren't able to do a comprehensive cleaning.

I'd honestly hope for a much higher standard than that for a
FSDG-compliant distro, especially one that carries the GNU mark.

> I would oppose adding such a partly-cleaned kernel to Guix itself,

But you don't!  That's what you get when you jump the gun and use
outdated cleaning up scripts, without waiting for us to verify,
update and release them for a newer version.

> but I wanted to enable users who need to use some other branch of
> Linux on their own systems to make a best-effort cleaning.

Besides the likelihood of something going wrong, that seems like a
backwards goal for a distro that is not expected to as much as point
users at a non-Free package.

I'm sure that's not what you intend, but this arrangement, plus your
mention of hurriedly getting releases out, adds up to an incentive to
disable the deblobbing so as to get a faster build.  I hope you'll agree
that this is undesirable.  As for how to speed up builds without
sacrificing freedom, see below.


>>> It allows us to update to a new point version (which usually
>>> includes security fixes) more quickly, before the linux-libre
>>> project reacts.
>> 
>> Any attempt outrun the Linux-libre project and get updates out sooner
>> is unwise. While major new kernel releases will definitely require
>> updates to the cleanup scripts, even minor patched versions
>> occasionally require changes too. Updating to a new version prior to
>> the Linux-libre project having had time to review that new version and
>> determine if any updates are needed to the scripts risks introducing
>> freedom problems in the corresponding Guix version.

> In my experience, the deblob scripts are very rarely changed after the
> first few point releases of a stable release series.

My personal experience tells me otherwise.  5.7 had only one update at
.8; 5.6, at .6 and .16; 5.5, at .3, .11 and .19; 5.4, at .14, .18, .27,
.34 and .44; 5.3, at .4 and .11; 5.2 at .1, .3 and .11; 5.1 at .2, .18
and .20; 5.0 at .7 and .16.  What you describe was true only of 4.17,
4.10, 4.3, 3.13, 3.5, and 3.2, i.e. 6 out of the 50 major releases
starting at 3.0.

> I know this because I always check for updates to the deblob scripts
> whenever I update linux-libre in Guix.  In practice, the deblob scripts used 
> by
> Guix are never more than 1 or 2 micro versions behind the version of
> Linux they are applied to.

There have been 61 script updates for the 1274 4.*.*-gnu* and 5.*.*-gnu*
stable releases, so Guix has shipped potentially non-FSDG code, that
*would* have been flagged by deblob-check on the tarballs, at between 5%
and 10% of these releases.  Does that sound like a good standard for a
freedom-first distro to aim for?


>> The moment that the Linux-libre project determines that scripts are
>> suitable is the moment that the new cleaned-up release is ready to
>> publish in git and the appropriate tags will then appear in git. The
>> compressed tarballs come some time later.

> I prefer to avoid unnecessary delays when applying micro kernel updates,

Sorry, but it doesn't look like you do.  If you did, you would be taking
a cleaned up tree instead of re-deblobbing it.  You skip even the
automated verification we do, which saves you some time, but at what
price?

If you waited another 30 minutes for our cleaned-up and verified tree to
be available from git, you'd save yourself the 20 minutes of cleaning-up
and another 20 minutes of deblob-checking the tarball for known or
likely freedom issues.  That sounds like a net win to me.

Now, if your build machines clean up and verify much faster than ours,
I'd be pretty glad to use them to get the verified commits in place so
that you could use them faster.

> I also consider it unwise for all of us, as a matter of habit or policy,
> to trust the integrity of the computer systems used by the Linux-libre
> project to perform the deblobbing.

I welcome double-checking of our cleaning up at all levels, but why are
you setting a higher trust standard for us than for a project known to
be at odds with our shared goals, such as Linux?  You don't apply the
patches that went into it since the last known good release to
double-check their releases, do you?  For most projects, you just take
their tarballs or tags and build it.  For Linux-libre, you start from
(untrustworthy) Linux, run the (presumed untrustworthy) cleaning up
scripts, and blindly trust the result.  There's no self-verification run
with deblob-check, no compare with our release, nothing.

If you were to test the integrity of our releases, you'd think you'd at
least look at them.  Starting from a known-good Linux release and
applying patches to double-check the results is expensive, so it makes
sense to do that only occasionally, rather than as part of every build.
Deblobbing and checking the result is also expensive, so it also makes
sense for you to do so only occasionally, rather than as part of every
build.

But the point stands that, for someone who'd rather trust no one, you're
blindly trusting both Linux and Linux-libre.  The former when it comes
to base releases you don't check; the latter when it comes to scripts
whose results you hardly even look at.  Why not reduce your trust base
to just Linux-libre, and treat is as a citizen of the same class as
nearly every other project you build, and satisfy your trust-but-verify
needs looking into what changes between one of our releases and another?


#bisecting

You can even take one of our releases and apply the patches that went
into the next upstream stable release, and check that what you get
matches our own corresponding release.  Some 98% of the time, they will
be exact matches.  Occasionally, there will be a difference, and then
you'll likely find a corresponding change in the deblobbing scripts, or
a preexisting pattern that caused the change.  We do this for every
release, as part of our pre-release checks, and you're welcome to do so
as well, and to use the resulting tree to bisect problems.

You'll see the builds are much faster if you don't have to deblob every
build.

Now, we've long had plans to publish a cleaned-up repo with Linux git
history.  It would take a massive amount of work to get it started, but
after that, getting new releases out might be about as fast as running a
git merge.  Ok, not that fast because there'd be some checking, but you
get the idea.

Other ideas to speed up our release process are:

- enabling cleaning up in multiple concurrent processes, like 'make -j'

- ditto for the deblob-check tarball verification

- use faster machines for the above

- monitor upstream git and fire up cleaning up automatically, and move
the various manual signatures of commits, tarballs, and logs to the very
end, after manual checking

These would enable the git commits to be out sooner.  Currently, our
best case is to push a release to the git release archive about one hour
after the upstream release, but we can only get that realistically when
there's only one release to do.  Most often, there are four to seven
releases at once, and then, since we don't always react immediately (one
needs to sleep occasionally ;-) and we don't get early warnings,
especially about major security issues, and our main workhorse has
limited capacity, we end up at about one hour per release anyway, having
them all ready at about the same time.

(Compression of tarballs then takes another half hour or more per
release, but the releases are pushed to the git release archive long
before compression is completed)

-- 
Alexandre Oliva, happy hacker
https://FSFLA.org/blogs/lxo/
Free Software Activist
GNU Toolchain Engineer