Re: [PATCH v2 2/3] gitlab-ci.yml: Add jobs to build the EDK2 firmware binaries

[Laszlo Ersek]

On 01/07/20 16:31, Philippe Mathieu-Daudé wrote:
> Add two GitLab job to build the EDK2 firmware binaries.
> 
> The first job build a Docker image with the packages requisite
> to build EDK2, and store this image in the GitLab registry.
> The second job pull the image from the registry and build the
> EDK2 firmware binaries.
> 
> The docker image is only rebuilt if the GitLab YAML or the
> Dockerfile is updated.
> The second job is only built when the roms/edk2/ submodule is
> updated, when a git-ref starts with 'edk2' or when the last
> commit contains 'EDK2'. The files generated are archived in
> the artifacts.zip file.
> 
> With edk2-stable201905, it took 2 minutes 52 seconds to build
> the docker image, and 36 minutes 28 seconds to generate the
> artifacts.zip with the firmware binaries (filesize: 10MiB).
> 
> See: https://gitlab.com/philmd/qemu/pipelines/107553178
> 
> Signed-off-by: Philippe Mathieu-Daudé <address@hidden>
> ---
> v2:
> - Split job, build docker image first
> - addressed Laszlo comments but kept the git-submodule-update call
> ---
>  .gitlab-ci-edk2.yml          | 49 ++++++++++++++++++++++++++++++++++++
>  .gitlab-ci.d/edk2/Dockerfile | 27 ++++++++++++++++++++
>  .gitlab-ci.yml               |  3 +++
>  MAINTAINERS                  |  4 ++-
>  4 files changed, 82 insertions(+), 1 deletion(-)
>  create mode 100644 .gitlab-ci-edk2.yml
>  create mode 100644 .gitlab-ci.d/edk2/Dockerfile
> 
> diff --git a/.gitlab-ci-edk2.yml b/.gitlab-ci-edk2.yml
> new file mode 100644
> index 0000000000..088ba4b43a
> --- /dev/null
> +++ b/.gitlab-ci-edk2.yml
> @@ -0,0 +1,49 @@
> +docker-edk2:
> + stage: build
> + rules: # Only run this job when the Dockerfile is modified
> + - changes:
> +   - .gitlab-ci-edk2.yml
> +   - .gitlab-ci.d/edk2/Dockerfile
> +   when: always
> + image: docker:19.03.1
> + services:
> + - docker:19.03.1-dind
> + variables:
> +  GIT_DEPTH: 3
> +  IMAGE_TAG: $CI_REGISTRY_IMAGE:edk2-cross-build
> +  # We don't use TLS
> +  DOCKER_HOST: tcp://docker:2375
> +  DOCKER_TLS_CERTDIR: ""
> + before_script:
> + - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
> + script:
> + - docker pull $IMAGE_TAG || true
> + - docker build --cache-from $IMAGE_TAG --tag 
> $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
> +                                        --tag $IMAGE_TAG .gitlab-ci.d/edk2
> + - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
> + - docker push $IMAGE_TAG
> +
> +build-edk2:
> + rules: # Only run this job when ...
> + - changes: # ... roms/edk2/ is modified (submodule updated)
> +   - roms/edk2/*
> +   when: always
> + - if: '$CI_COMMIT_REF_NAME =~ /^edk2/' # or the branch/tag starts with 
> 'edk2'
> +   when: always
> + - if: '$CI_COMMIT_MESSAGE =~ /edk2/i' # or last commit description contains 
> 'EDK2'
> +   when: always
> + artifacts:
> +   paths: # 'artifacts.zip' will contains the following files:
> +   - pc-bios/edk2*bz2
> +   - pc-bios/edk2-licenses.txt
> +   - edk2-stdout.log
> +   - edk2-stderr.log
> + image: $CI_REGISTRY_IMAGE:edk2-cross-build
> + variables:
> +   GIT_DEPTH: 3
> + script: # Clone the required submodules and build EDK2
> + - git submodule update --init roms/edk2
> + - git -C roms/edk2 submodule update --init
> + - export JOBS=$(($(getconf _NPROCESSORS_ONLN) + 1))
> + - echo "=== Using ${JOBS} simultaneous jobs ==="
> + - make -j${JOBS} -C roms efi 2>&1 1>edk2-stdout.log | tee -a 
> edk2-stderr.log >&2
> diff --git a/.gitlab-ci.d/edk2/Dockerfile b/.gitlab-ci.d/edk2/Dockerfile
> new file mode 100644
> index 0000000000..b4584d1cf6
> --- /dev/null
> +++ b/.gitlab-ci.d/edk2/Dockerfile
> @@ -0,0 +1,27 @@
> +#
> +# Docker image to cross-compile EDK2 firmware binaries
> +#
> +FROM ubuntu:16.04
> +
> +MAINTAINER Philippe Mathieu-Daudé <address@hidden>
> +
> +# Install packages required to build EDK2
> +RUN apt update \
> +    && \
> +    \
> +    DEBIAN_FRONTEND=noninteractive \
> +    apt install --assume-yes --no-install-recommends \
> +        build-essential \
> +        ca-certificates \
> +        dos2unix \
> +        gcc-aarch64-linux-gnu \
> +        gcc-arm-linux-gnueabi \
> +        git \
> +        iasl \
> +        make \
> +        nasm \
> +        python \
> +        uuid-dev \
> +    && \
> +    \
> +    rm -rf /var/lib/apt/lists/*
> diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml
> index ebcef0ebe9..f799246047 100644
> --- a/.gitlab-ci.yml
> +++ b/.gitlab-ci.yml
> @@ -1,3 +1,6 @@
> +include:
> +  - local: '/.gitlab-ci-edk2.yml'
> +
>  before_script:
>   - apt-get update -qq
>   - apt-get install -y -qq flex bison libglib2.0-dev libpixman-1-dev 
> genisoimage
> diff --git a/MAINTAINERS b/MAINTAINERS
> index cd2dc137a3..93620ed406 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -2358,6 +2358,8 @@ F: roms/edk2
>  F: roms/edk2-*
>  F: tests/data/uefi-boot-images/
>  F: tests/uefi-test-tools/
> +F: .gitlab-ci-edk2.yml
> +F: .gitlab-ci.d/edk2/
>  
>  Usermode Emulation
>  ------------------
> @@ -2701,7 +2703,7 @@ W: https://cirrus-ci.com/github/qemu/qemu
>  GitLab Continuous Integration
>  M: Thomas Huth <address@hidden>
>  S: Maintained
> -F: .gitlab-ci.yml
> +F: .gitlab-ci*.yml
>  
>  Guest Test Compilation Support
>  M: Alex Bennée <address@hidden>
> 

The Docker stuff is not my cup of tea, but the edk2 build stuff looks OK.

Reviewed-by: Laszlo Ersek <address@hidden>

Thanks
Laszlo