Re: backdoor injection via release tarballs combined with binary artifacts (was Re: Backdoor in upstream xz-utils)

[Giovanni Biscuolo]

Hi Attila and guix-security team,

Attila Lendvai <attila@lendvai.name> writes:

>> Are really "configure scripts containing hundreds of thousands of lines
>> of code not present in the upstream VCS" the norm?
>
> pretty much for all C and C++ projects that use autoconf... which is
> numerous, especially among the core GNU components.

OK, thank you for the confirmation.

[...]

>> ...or is it better to completely avoid release tarballs as our sources
>> uris?
>
> yes, and this^ would guarantee the previous point, but it's not always 
> trivial.
>
> as an example see this: https://issues.guix.gnu.org/61750

[...]

> it breaks crosscompilation, because the host cannot execute the target
> binary.

OK thanks, I missed that.

In general, there is really no other solution for projects than to
distribute some artifacts "out of band" or renounce to crosscompile?!?

Are there other issues (different from the "host cannot execute target
binary") that makes relesase tarballs indispensable for some upstream
projects?

AFAIU the only thing that /could/ "save" source tarballs it's their
/scientific/ reproducibility.  In this direction there is a very
interesting patchset from Janneke Nieuwenhuizen to try to get a
reproducible _Guix_ release tarball:

https://issues.guix.gnu.org/70169
«Reproducible `make dist' tarball in defiance of Autotools and Gettext»

Obviously having a reproducible tarball makes _practical_ the
"pragmatically impossible" task to reproduce a release tarball to check
if it corresponds to the same **build** (make dist) performed in the
official DVCS repo; only this could "save" all the "build software using
release tarball" workflow.

...but /in general/ here we are _downstream_, we have absolutely no
control over upstream, and it's _very_ unlikely that we'll see a *good*
solution to the tarball reproduciblity problem applied "in the wild
upstream" soon.

I said "a **good* solution" because some proposals I'm reading about are
/bad/ _complications_ that absolutely are NOT really solving the source
tarball reproduciblity problem [1]; for example:

1. build the tarball on the RM host using a docker container
(unreproducible built) and call it "a reproducible release tarball":
https://medium.com/@lanoxx/creating-reproducible-release-tarballs-fa2e2ce745a7

2. have a CI system based on github actions [2] and call it "fully
verifiable": https://externals.io/message/122811#122814 (from
php.internals mailing list)

So, while "almost all the world" is applying _wrong_ solutions to the
source tarball reproducibility problem, what can Guix do?

Even if We™ (ehrm) find a solution to the source tarball reproducibility
problem (potentially allowing us to patch all the upstream makefiles
with specific phases in our packages definitions) are we really going to
start our own (or one managed by the reproducible build community)
"reproducible source tarballs" repository?  Is this feaseable?

I think there is no solution that can "pragmatically save" the source
tarballs of all the software packaged in Guix (and all other
distributions part of the reproducible builds effort).

> but these generated man files are part of the release tarball, so
> cross compilation works fine using the tarball.

AFAIU *in this case* there is an easy alternative: distribute the
(generated) man files as *code* tracked in the DVCS (e.g. git) repo
itself.  IMHO it's likely that this workflow can fix most if not all the
crosscompilation issues, no?

In general, AFAIU it's against reproducibility to distribute
pre-generated (compiled? transpiled?) artifacts in a tarball that are
not present in the official DVCS repo, especially when tarballs are
_not_ reproducible (and they are not in likely 99.9% of cases).

> all in all, just by following my gut insctincts, i was advodating for
> building everything from git even before the exposure of this
> backdoor. in fact, i found it surprising as a guix newbie that not
> everything is built from git (or their VCS of choice).

Given the current situation so clearly exposed by the "xz backdoor"
case, this is something Guix should seriously consider.

I mean: Guix should seriously consider to drop source tarballs and
_also_ all pre-compiled artifacts distributed only via that tarballs.

I don't like this proposal, but I see no other "pragmatically possible"
solution.

AFAIU no need to rush, but I'm afraid that the class of attacks we can
call "supply-chain backdoor injection due to source tarball
pragmatically impossible verifiability" are hard to deploy but
unfortunately not _too_ hard.

[...]

Thanks! Gio'


[1] this boils down to the unfortunate fact that "reproducibility" is a
very misunderstood concept [1.1], even by some very skilled (experienced?)
programmers

[1.1] because it's strictly related to good _redistribution_ of
_trusted_ software, not to good programming

[2]
https://docs.github.com/en/actions/learn-github-actions/understanding-github-actions#runners
«each workflow run executes in a fresh, newly-provisioned virtual machine.»
see also
https://www.paloaltonetworks.com/blog/prisma-cloud/unpinnable-actions-github-security/
for security concerns about GitHub actions relying on Docker containers
used for "reproducibility" purposes.

-- 
Giovanni Biscuolo

Xelera IT Infrastructures

signature.asc
Description: PGP signature