bug#42162: Recovering source tarballs

[Timothy Sample]

Hey,

Ludovic Courtès <ludo@gnu.org> writes:

> Note that we have <https://guix.gnu.org/sources.json>.  Last I checked,
> SWH was ingesting it in its “qualification” instance, so it should be
> ingesting it for good real soon if it’s not doing it already.

Oh fantastic!  I was going to volunteer to do it, so that’s one thing
off my list.

> One can easily write a procedure that takes a tarball and returns a
> <computed-file> that builds its database entry.  So at each commit, we’d
> just rebuild things that have changed.

That makes more sense.  I will give this a shot soon.

> If we expose the database over HTTP (like over cgit), we can arrange so
> that (guix download) simply GETs db.example.org/sha256/xyz.  No need to
> fetch the whole database.
>
> It might be more reasonable to have a real database and a real service
> around it, I’m sure Chris Baines would agree ;-), but we can choose URLs
> that could easily be implemented by a “real” service instead of cgit in
> the future.

I got it working over cgit shortly after sending my last message.  :)  So
far, I am very much on team “good enough for now”.

> Timothy Sample <samplet@ngyro.com> skribis:
>
>> I was imagining an escape hatch beyond this, where one could look up a
>> provenance record from when Disarchive ingested and verified a source
>> code archive.  The provenance record would tell you which version of
>> Guix was used when saving the archive, so you could try your luck with
>> using “guix time-machine” to reproduce Disarchive’s original
>> computation.  If we perform database migrations, you would need to
>> travel back in time in the database, too.  The idea is that you could
>> work around breakages in Disarchive automatically using the Power of
>> Guix™.  Just a stray thought, really.
>
> Seems to me it Shouldn’t Be Necessary?  :-)
>
> I mean, as long as the format is extensible and “future-proof”, we’ll
> always be able to rebuild tarballs and then re-disassemble them if we
> need to compute new hashes or whatever.

If Disarchive relies on external compressors, there’s an outside chance
that those compressors could change under our feet.  In that case, one
would want to be able to track down exactly which version of XZ was used
when Disarchive verified that it could reassemble a given source
archive.  Maybe I’m being paranoid, but if the database entries are
being computed by the CI infrastructure it would be pretty easy to note
the Guix commit just in case.

> I was thinking that it might be best to not use Guix for computations.
> For example, have “disarchive save” not build derivations and instead do
> everything “here and now”.  That would make it easier for others to
> adopt.  Wait, looking at the Git history, it looks like you already
> addressed that point, neat.  :-)

Since my last message I managed to remove Guix as dependency completely.
Right now it loads ‘(guix swh)’ opportunistically, but I might just copy
the code in.  Directory references now support multiple “addresses” so
that you could have Nix-style, SWH-style, IPFS-style, etc.  Hopefully my
next message will have a WIP patch enabling Guix to use Disarchive!


-- Tim