Re: CI is not processing jobs

[Ludovic Courtès]

Hi Guix!

Ludovic Courtès <ludo@gnu.org> skribis:

> Ludovic Courtès <ludo@gnu.org> skribis:
>
>> The longer story is that Ricardo noticed that the build backlog had been
>> growing for a couple of months (see “Pending builds”):
>>
>>   https://ci.guix.gnu.org/metrics
>>
>> We discussed this on guix-sysadmin and found that this was due to the
>> poor performance of a SQL query at the core of ‘cuirass remote-server’
>> that was roughly linear in the number of packages in the database.  With
>> help from Chris Baines, this is now fixed:
>>
>>   
>> https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/commit/?id=17338588d4862b04e9e405c1244a2ea703b50d98
>
> This is now deployed on ci.guix.gnu.org.

This has been running for ~4 days now; the number of pending builds has
significantly decreased (in particular, you’ll be delighted to get
substitutes for ‘core-updates’!):

  https://ci.guix.gnu.org/metrics

Almost all the x86 builds have been consumed:

--8<---------------cut here---------------start------------->8---
cuirass=> select count(*) from builds where status = -2 and system 
='x86_64-linux';
 count 
-------
   748
(1 row)

cuirass=> select count(*) from builds where status = -2 and system 
='i686-linux';
 count 
-------
     0
(1 row)

cuirass=> select count(*) from builds where status = -2 and system 
='powerpc64le-linux';
 count  
--------
 110963
(1 row)

cuirass=> select count(*) from builds where status = -2 ;
 count  
--------
 382892
(1 row)
--8<---------------cut here---------------end--------------->8---

The “Pending builds” plot above shows we’re reaching a plateau: this is
because the 382,000+ remaining builds are non-x86_64 and we lack
resources for these platforms (Arm in particular).

We should probably investigate and cancel builds that correspond to old
evaluations or now-irrelevant jobsets.

Ludo’.