Re: Serious doubts about Gitlab CI

[Daniel P . Berrangé]

On Tue, Mar 30, 2021 at 01:55:48PM +0200, Thomas Huth wrote:
> On 30/03/2021 13.19, Daniel P. Berrangé wrote:

> > Another example, is that we test builds on centos7 with
> > three different combos of crypto backend settings. This was
> > to exercise bugs we've seen in old crypto packages in RHEL-7
> > but in reality, it is probably overkill, because downstream
> > RHEL-7 only cares about one specific combination.
> 
> Care to send a patch? Or shall we just wait one more months and then remove
> these jobs (since we won't support RHEL7 after QEMU 6.0 anymore)?

Yeah, we'll be able to cull this entirely very soon, including
both the C backcompat code and CI jobs at the same time, so I'll
just wait.


> > Our docker containers install ccache already and I could have sworn
> > that we use that in gitlab, but now I'm not so sure. We're only
> > saving the "build/" directory as an artifact between jobs, and I'm
> > not sure that directory holds the ccache cache.
> 
> AFAIK we never really enabled ccache in the gitlab-CI, only in Travis.
> 
> > > This is as far as I've gotten with thinking about CI efficiency. Do you
> > > think these optimizations are worth investigating or should we keep it
> > > simple and just disable many builds by default?
> > 
> > ccache is a no-brainer and assuming it isn't already working with
> > our gitlab jobs, we must fix that asap.
> 
> I've found some nice instructions here:
> 
> https://gould.cx/ted/blog/2017/06/10/ccache-for-Gitlab-CI/
> 
> ... and just kicked off a build with these modifications, let's see how it
> goes...

Yep, that looks similar to what we do in libvirt, though we don't override
the compiler at the job level. Instead we just ensure the dir containing
ccache symlinks appears first in $PATH.

So in containers we have this:

https://gitlab.com/libvirt/libvirt/-/blob/master/ci/containers/centos-8.Dockerfile

and in gitlab-ci.yml we have env vars set

  export CCACHE_BASEDIR="$(pwd)"
  export CCACHE_DIR="$CCACHE_BASEDIR/ccache"
  export CCACHE_MAXSIZE="500M"
  export PATH="$CCACHE_WRAPPERSDIR:$PATH"

And per-job caches:

  cache:
    paths:
      - ccache/
    key: "$CI_JOB_NAME"

note the "key" is important to avoid clashing caches from different
envs.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|