Re: Serious doubts about Gitlab CI

[Stefan Hajnoczi]

On Fri, Mar 19, 2021 at 12:27:10PM -0300, Wainer dos Santos Moschetta wrote:
> Hi,
> 
> On 3/19/21 8:34 AM, Philippe Mathieu-Daudé wrote:
> > On 3/19/21 11:59 AM, Paolo Bonzini wrote:
> > > On 19/03/21 11:18, Andrew Jones wrote:
> > > > > Yikes, that is 41 hours per CI run. I wonder if GitLab's CI minutes 
> > > > > are
> > > > > on slow machines or if we'll hit the same issue with dedicated 
> > > > > runners.
> > > > > It seems like CI optimization will be necessary...
> > > > > 
> > > > We need to reduce the amount of CI we do, not only because we can't
> > > > afford
> > > > it, but because it's wasteful. I hate to think of all the kWhs spent
> > > > testing the exact same code in the exact same way, since everyone runs
> > > > everything with a simple 'git push'.
> > > Yes, I thought the same.
> > > 
> > > > IMHO, 'git push' shouldn't trigger
> > > > anything. Starting CI should be an explicit step.
> > * tests/acceptance: Only run tests tagged 'gating-ci' on GitLab CI
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg756464.html
> > 
> > * gitlab-ci: Allow forks to select & restrict build jobs
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg758331.html
> 
> In my opinion that series is the first step towards a smart CI. It got some
> reviews of Thomas and myself already but it didn't move ahead. If Philippe
> for some reason cannot continue that work, I'm volunteering to take it over.

Hi,
I wanted to follow up with a summary of the CI jobs:

1. Containers & Containers Layer2 - ~3 minutes/job x 39 jobs
2. Builds - ~50 minutes/job x 61 jobs
3. Tests - ~12 minutes/job x 20 jobs
4. Deploy - 52 minutes x 1 job

The Builds phase consumes the most CI minutes. If we can optimize this
phase then we'll achieve the biggest impact.

In the short term builds could be disabled. However, in the long term I
think full build coverage is desirable to prevent merging code that
breaks certain host OSes/architectures (e.g. stable Linux distros,
macOS, etc).

Traditionally ccache (https://ccache.dev/) was used to detect
recompilation of the same compiler input files. This is trickier to do
in GitLab CI since it would be necessary to share and update a cache,
potentially between untrusted users. Unfortunately this shifts the
bottleneck from CPU to network in a CI-as-a-Service environment since
the cached build output needs to be accessed by the linker on the CI
runner but is stored remotely.

A complementary approach is avoiding compilation altogether when code
changes do not affect a build target. For example, a change to
qemu-storage-daemon.c does not require rebuilding the system emulator
targets. Either the compiler or the build system could produce a
manifest of source files that went into a build target, and that
information is what's needed to avoid compiling unchanged targets.

Ideally the CI would look at the code changes and only launch jobs that
were affected. Those jobs would use a C compiler cache to avoid
rebuilding compiler input that has not changed. Basically, we need
incremental builds.

This is as far as I've gotten with thinking about CI efficiency. Do you
think these optimizations are worth investigating or should we keep it
simple and just disable many builds by default?

Stefan

signature.asc
Description: PGP signature