Re: Serious doubts about Gitlab CI

[Philippe Mathieu-Daudé]

On 3/30/21 1:55 PM, Thomas Huth wrote:
> On 30/03/2021 13.19, Daniel P. Berrangé wrote:
>> On Mon, Mar 29, 2021 at 03:10:36PM +0100, Stefan Hajnoczi wrote:

>>> Traditionally ccache (https://ccache.dev/) was used to detect
>>> recompilation of the same compiler input files. This is trickier to do
>>> in GitLab CI since it would be necessary to share and update a cache,
>>> potentially between untrusted users. Unfortunately this shifts the
>>> bottleneck from CPU to network in a CI-as-a-Service environment since
>>> the cached build output needs to be accessed by the linker on the CI
>>> runner but is stored remotely.
>>
>> Our docker containers install ccache already and I could have sworn
>> that we use that in gitlab, but now I'm not so sure. We're only
>> saving the "build/" directory as an artifact between jobs, and I'm
>> not sure that directory holds the ccache cache.
> 
> AFAIK we never really enabled ccache in the gitlab-CI, only in Travis.

Back then the Travis setup was simpler, and it took me 2 to 3 weeks
to get it right (probably spending 3 to 4h a day on it).

>>> This is as far as I've gotten with thinking about CI efficiency. Do you
>>> think these optimizations are worth investigating or should we keep it
>>> simple and just disable many builds by default?
>>
>> ccache is a no-brainer and assuming it isn't already working with
>> our gitlab jobs, we must fix that asap.
> 
> I've found some nice instructions here:
> 
> https://gould.cx/ted/blog/2017/06/10/ccache-for-Gitlab-CI/
> 
> ... and just kicked off a build with these modifications, let's see how
> it goes...

But we cross-build in Docker containers, so you need to mount the
cache dir in the container and set the CCACHE_DIR env var, isn't it?

Watch out about custom runners. If we do too many changes on the
free-tier runners, we'll never have the custom runner series integrated.

My 2 cents.

Regards,

Phil.