More progress with the Guix Data Service

[Christopher Baines]

Hey,

In summary, I think the Guix Data Service might be getting useful enough
that setting it up properly might be a good next step, and I'd be
interested in what others think?

A bit over a month ago, I sent out an update about one of the things
I've been working on, something I've been calling the "Guix Data
Service" [1].

1: https://lists.gnu.org/archive/html/guix-devel/2019-04/msg00094.html

Since then, I've made some more progress. There's a new statistics page
[2]. I've got used to using Sqitch [3] to help manage the database
schema, and I've added some basic tests.

2: https://prototype-guix-data-service.cbaines.net/statistic
3: http://guix.gnu.org/packages/sqitch-0.9999/

The error handling for loading new revisions is also more resilient now.

As well as listening to the Guix Commits mailing list for emails about
new revisions, more of the information in these emails is now stored, in
particular, the time they were sent, and the branch the email applies
to. This can be seen on the new Branches page [4].

4: https://prototype-guix-data-service.cbaines.net/branches

The content negotiation a little bit, at least in terms of the code, and
JSON output support has been added to more pages.

There's now a basic search function on the packages page [5], and the
location, and the licenses for packages is now being stored (which can
be seen on the page for a package, for example [6]).

5: 
https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages?search_query=Guile&field=version&field=synopsis&after_name=&limit_results=1000
6: 
https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/package/0ad/0.0.23b-alpha

The location and license information is something I added specifically,
as I noticed the Repology [7] service scraped [8] these from the Guix
website.

7: https://repology.org/
8: 
https://github.com/repology/repology/blob/master/repology/fetchers/fetchers/guix.py

While the Guix Data Service started as something to enable better
understanding patches in an automated way, I think there are more uses
for it, and initially, it's probably better to focus on the simple ones.

The Repology use case is pretty simple, I think ideally there would be
some machine readable data about the current state of packages in Guix
available over the internet, and Repology would be able to download that
on a regular basis.

The URL is a bit long, but I think that is now close to being possible
with the Guix data service. I haven't got something working yet to
easily access data for the latest revision, but for a particular
revision, you can request a JSON file containing all the information I
think Repology currently gets about all packages. For example:

  
https://prototype-guix-data-service.cbaines.net/revision/f52e83470b05b2473ea13feb2842a1330c316a00/packages.json?field=version&field=synopsis&field=description&field=home-page&field=location&field=licenses&limit_results=99999

This is just the software side of the problem though. If this was to be
used by Repology, it would have to be a more permanent thing, similar to
the Cuirass and Mumi services that are currently setup around Guix. Does
anyone have any thoughts on this?

Thanks,

Chris

signature.asc
Description: PGP signature