Re: [PATCH 7/8] tests/vm: Added a new script for ubuntu.aarch64.

[Alex Bennée]

Robert Foley <address@hidden> writes:

> ubuntu.aarch64 provides a script to create an Ubuntu 18.04 VM.
> Another new file is also added aarch64vm.py, which is a module with
> common methods used by aarch64 VMs, such as how to create the
> flash images.
>
> Signed-off-by: Robert Foley <address@hidden>
> Reviewed-by: Peter Puhov <address@hidden>
> ---
>  tests/vm/Makefile.include |   7 +-
>  tests/vm/aarch64vm.py     |  41 +++++++++++
>  tests/vm/ubuntu.aarch64   | 144 ++++++++++++++++++++++++++++++++++++++
>  3 files changed, 190 insertions(+), 2 deletions(-)
>  create mode 100644 tests/vm/aarch64vm.py
>  create mode 100755 tests/vm/ubuntu.aarch64
>
> diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
> index 9e7c46a473..966b417ba7 100644
> --- a/tests/vm/Makefile.include
> +++ b/tests/vm/Makefile.include
> @@ -2,7 +2,7 @@
>  
>  .PHONY: vm-build-all vm-clean-all
>  
> -IMAGES := ubuntu.i386 freebsd netbsd openbsd centos fedora
> +IMAGES := ubuntu.i386 freebsd netbsd openbsd centos fedora ubuntu.aarch64
>  IMAGES_DIR := $(HOME)/.cache/qemu-vm/images
>  IMAGE_FILES := $(patsubst %, $(IMAGES_DIR)/%.img, $(IMAGES))
>  
> @@ -18,6 +18,7 @@ vm-help vm-test:
>       @echo "  vm-build-openbsd                - Build QEMU in OpenBSD VM"
>       @echo "  vm-build-centos                 - Build QEMU in CentOS VM, 
> with Docker"
>       @echo "  vm-build-fedora                 - Build QEMU in Fedora VM"
> +     @echo "  vm-build-ubuntu.aarch64         - Build QEMU in ubuntu aarch64 
> VM"
>       @echo ""
>       @echo "  vm-build-all                    - Build QEMU in all VMs"
>       @echo "  vm-clean-all                    - Clean up VM images"
> @@ -35,6 +36,8 @@ vm-help vm-test:
>       @echo "    V=1                           - Enable verbose ouput on host 
> and guest commands"
>       @echo "    QEMU=/path/to/qemu            - Change path to QEMU binary"
>       @echo "    QEMU_IMG=/path/to/qemu-img    - Change path to qemu-img tool"
> +     @echo "    QEMU_CONFIG=/path/conf.yml    - Change path to VM 
> configuration .yml file."
> +     @echo "                                    See config_example.yml for 
> file format details."
>  
>  vm-build-all: $(addprefix vm-build-, $(IMAGES))
>  
> @@ -80,7 +83,7 @@ vm-boot-serial-%: $(IMAGES_DIR)/%.img
>  
>  vm-boot-ssh-%: $(IMAGES_DIR)/%.img
>       $(call quiet-command, \
> -             $(SRC_PATH)/tests/vm/$* \
> +             $(PYTHON) $(SRC_PATH)/tests/vm/$* \

This seems like it should be in a different patch.

>               $(if $(J),--jobs $(J)) \
>               --image "$<" \
>               --interactive \
> diff --git a/tests/vm/aarch64vm.py b/tests/vm/aarch64vm.py
> new file mode 100644
> index 0000000000..43f841571f
> --- /dev/null
> +++ b/tests/vm/aarch64vm.py
> @@ -0,0 +1,41 @@
> +#!/usr/bin/env python
> +#
> +# VM testing aarch64 library
> +#
> +# Copyright 2020 Linaro
> +#
> +# Authors:
> +#  Robert Foley <address@hidden>
> +#
> +# This code is licensed under the GPL version 2 or later.  See
> +# the COPYING file in the top-level directory.
> +#
> +import os
> +import sys
> +import subprocess
> +import basevm
> +
> +
> +def create_flash_images():
> +    """Creates the appropriate pflash files
> +       for an aarch64 VM."""
> +    subprocess.check_call(["dd", "if=/dev/zero", "of=flash0.img",
> +                           "bs=1M", "count=64"])
> +    # A reliable way to get the QEMU EFI image is via an installed package.
> +    efi_img = "/usr/share/qemu-efi-aarch64/QEMU_EFI.fd"
> +    if not os.path.exists(efi_img):
> +        sys.stderr.write("*** {} is missing\n".format(efi_img))
> +        sys.stderr.write("*** please install qemu-efi-aarch64 package\n")
> +        exit(3)
> +    subprocess.check_call(["dd", "if={}".format(efi_img),
> +                           "of=flash0.img", "conv=notrunc"])
> +    subprocess.check_call(["dd", "if=/dev/zero",
> +                           "of=flash1.img", "bs=1M", "count=64"])
> +
> +def get_pflash_args():
> +    """Returns a string that can be used to
> +       boot qemu using the appropriate pflash files
> +       for aarch64."""
> +    pflash_args = "-drive file=flash0.img,format=raw,if=pflash "\
> +                  "-drive file=flash1.img,format=raw,if=pflash"
> +    return pflash_args.split(" ")
> diff --git a/tests/vm/ubuntu.aarch64 b/tests/vm/ubuntu.aarch64
> new file mode 100755
> index 0000000000..941f7f5166
> --- /dev/null
> +++ b/tests/vm/ubuntu.aarch64
> @@ -0,0 +1,144 @@
> +#!/usr/bin/env python
> +#
> +# Ubuntu aarch64 image
> +#
> +# Copyright 2020 Linaro
> +#
> +# Authors:
> +#  Robert Foley <address@hidden>
> +#  Originally based on ubuntu.i386 Fam Zheng <address@hidden>
> +#
> +# This code is licensed under the GPL version 2 or later.  See
> +# the COPYING file in the top-level directory.
> +#
> +
> +import os
> +import sys
> +import subprocess
> +import basevm
> +import time
> +import aarch64vm
> +
> +DEFAULT_CONFIG = {
> +    'cpu'          : "max",
> +    'machine'      : "virt,gic-version=max",

According to virt.c:

  Valid values are 2, 3 and host

but max should work on TCG. However we need a more recent QEMU than my
system one for it to work. Otherwise you see:

  DEBUG:qemu.machine:Error launching VM
  DEBUG:qemu.machine:Command: 'qemu-system-aarch64 -display none -vga none 
-chardev socket,id=mon,path=/var/tmp/qemu-18443-monitor.sock -mon 
chardev=mon,mode=control -machine
  virt,gic-version=max -chardev 
socket,id=console,path=/var/tmp/qemu-18443-console.sock,server,nowait -serial 
chardev:console -nodefaults -m 4G -cpu max -netdev 
user,id=vnet,hostfwd=:127.0.0.1:0-:22 -device virtio-net-pci,netdev=vnet -vnc 
127.0.0.1:0,to=20 -drive 
file=/home/alex.bennee/.cache/qemu-vm/images/ubuntu.aarch64.img.tmp,if=none,id=drive0,cache=writeback
 -device virtio-blk,drive=drive0,bootindex=0 -cdrom 
/home/alex.bennee/lsrc/qemu.git/builds/all/vm-test-v8jttvl3.tmp/cloud-init.iso 
-drive file=flash0.img,format=raw,if=pflash -drive 
file=flash1.img,format=raw,if=pflash -accel tcg,thread=multi -smp 8 -device VGA'
  DEBUG:qemu.machine:Output: 'qemu-system-aarch64: Invalid gic-version 
value\nValid values are 3, 2, host.\n'
  ERROR:root:Failed to launch QEMU, command line:
  ERROR:root:qemu-system-aarch64 -nodefaults -m 4G -cpu max -netdev 
user,id=vnet,hostfwd=:127.0.0.1:0-:22 -device virtio-net-pci,netdev=vnet -vnc 
127.0.0.1:0,to=20 -drive 
file=/home/alex.bennee/.cache/qemu-vm/images/ubuntu.aarch64.img.tmp,if=none,id=drive0,cache=writeback
 -device virtio-blk,drive=drive0,bootindex=0 -cdrom 
/home/alex.bennee/lsrc/qemu.git/builds/all/vm-test-v8jttvl3.tmp/cloud-init.iso 
-drive file=flash0.img,format=raw,if=pflash -drive 
file=flash1.img,format=raw,if=pflash -accel tcg,thread=multi -smp 8 -device VGA
  ERROR:root:Log:
  ERROR:root:qemu-system-aarch64: Invalid gic-version value
  Valid values are 3, 2, host.

  ERROR:root:QEMU version >= 2.10 is required
  Failed to prepare guest environment
  Traceback (most recent call last):
    File "/home/alex.bennee/lsrc/qemu.git/tests/vm/basevm.py", line 514, in main
      return vm.build_image(args.image)
    File "/home/alex.bennee/lsrc/qemu.git/tests/vm/ubuntu.aarch64", line 114, 
in build_image
      self.boot(img_tmp, extra_args = ["-cdrom", ci_img])
    File "/home/alex.bennee/lsrc/qemu.git/tests/vm/ubuntu.aarch64", line 105, 
in boot
      super(UbuntuAarch64VM, self).boot(img, extra_args=extra_args)
    File "/home/alex.bennee/lsrc/qemu.git/tests/vm/basevm.py", line 246, in boot
      guest.launch()
    File 
"/home/alex.bennee/lsrc/qemu.git/tests/vm/../../python/qemu/machine.py", line 
302, in launch
      self._launch()
    File 
"/home/alex.bennee/lsrc/qemu.git/tests/vm/../../python/qemu/machine.py", line 
329, in _launch
      self._post_launch()
    File 
"/home/alex.bennee/lsrc/qemu.git/tests/vm/../../python/qemu/machine.py", line 
274, in _post_launch
      self._qmp.accept()
    File "/home/alex.bennee/lsrc/qemu.git/tests/vm/../../python/qemu/qmp.py", 
line 157, in accept
      return self.__negotiate_capabilities()
    File "/home/alex.bennee/lsrc/qemu.git/tests/vm/../../python/qemu/qmp.py", 
line 73, in __negotiate_capabilities
      raise QMPConnectError
  qemu.qmp.QMPConnectError
  /home/alex.bennee/lsrc/qemu.git/tests/vm/Makefile.include:51: recipe for 
target '/home/alex.bennee/.cache/qemu-vm/images/ubuntu.aarch64.img' failed
  make: *** [/home/alex.bennee/.cache/qemu-vm/images/ubuntu.aarch64.img] Error 2


> +    'install_cmds' : "apt-get update,"\
> +                     "apt-get build-dep -y qemu,"\
> +                     "apt-get install -y libfdt-dev flex bison",
> +    # We increase beyond the default time since during boot
> +    # it can take some time (many seconds) to log into the VM
> +    # especially using softmmu.
> +    'ssh_timeout'  : 60,
> +}
> +
> +class UbuntuAarch64VM(basevm.BaseVM):
> +    name = "ubuntu.aarch64"
> +    arch = "aarch64"
> +    image_name = "ubuntu-18.04-server-cloudimg-arm64.img"
> +    image_link = "https://cloud-images.ubuntu.com/releases/18.04/release/"; + 
> image_name
> +    login_prompt = "ubuntu-guest login:"
> +    BUILD_SCRIPT = """
> +        set -e;
> +        cd $(mktemp -d);
> +        sudo chmod a+r /dev/vdb;
> +        tar --checkpoint=.10 -xf /dev/vdb;
> +        ./configure {configure_opts};
> +        make --output-sync {target} -j{jobs} {verbose};
> +    """
> +    def _gen_cloud_init_iso(self):
> +        cidir = self._tmpdir
> +        mdata = open(os.path.join(cidir, "meta-data"), "w")
> +        mdata.writelines(["instance-id: ubuntu-vm-0\n",
> +                          "local-hostname: ubuntu-guest\n"])
> +        mdata.close()
> +        udata = open(os.path.join(cidir, "user-data"), "w")
> +        print("guest user:pw {}:{}".format(self.GUEST_USER, self.GUEST_PASS))
> +        udata.writelines(["#cloud-config\n",
> +                          "chpasswd:\n",
> +                          "  list: |\n",
> +                          "    root:%s\n" % self.ROOT_PASS,
> +                          "    %s:%s\n" % (self.GUEST_USER, self.GUEST_PASS),
> +                          "  expire: False\n",
> +                          "users:\n",
> +                          "  - name: %s\n" % self.GUEST_USER,
> +                          "    sudo: ALL=(ALL) NOPASSWD:ALL\n",
> +                          "    ssh-authorized-keys:\n",
> +                          "    - %s\n" % self.ssh_pub_key,
> +                          "  - name: root\n",
> +                          "    ssh-authorized-keys:\n",
> +                          "    - %s\n" % self.ssh_pub_key,
> +                          "locale: en_US.UTF-8\n"])
> +        proxy = os.environ.get("http_proxy")
> +        if not proxy is None:
> +            udata.writelines(["apt:\n",
> +                              "  proxy: %s" % proxy])
> +        udata.close()
> +        subprocess.check_call(["genisoimage", "-output", "cloud-init.iso",
> +                               "-volid", "cidata", "-joliet", "-rock",
> +                               "user-data", "meta-data"],
> +                               cwd=cidir,
> +                               stdin=self._devnull, stdout=self._stdout,
> +                               stderr=self._stdout)
> +
> +        return os.path.join(cidir, "cloud-init.iso")

It seems this function is proliferating. It certainly seems common
enough to be basevm functionality.

> +
> +    def boot(self, img, extra_args=None):
> +        aarch64vm.create_flash_images()
> +        default_args = aarch64vm.get_pflash_args()
> +        if extra_args:
> +            extra_args.extend(default_args)
> +        else:
> +            extra_args = default_args
> +        # We always add these performance tweaks
> +        # because without them, we boot so slowly that we
> +        # can time out finding the boot efi device.
> +        if os.geteuid() != 0:
> +            extra_args.extend(["-accel", "tcg,thread=multi"])

Hmmm thread=multi should already be enabled by default where it is safe
to do so. Also what has it to do with euid?

> +        if '-smp' not in extra_args and \
> +           '-smp' not in self._config['extra_args'] and \
> +           '-smp' not in self._args:
> +            # Only add if not already there to give caller option to change 
> it.
> +            extra_args.extend(["-smp", "8"])
> +
> +        # We have overridden boot() since aarch64 has additional parameters.
> +        # Call down to the base class method.
> +        super(UbuntuAarch64VM, self).boot(img, extra_args=extra_args)
> +
> +    def build_image(self, img):
> +        os_img = self._download_with_cache(self.image_link)
> +        img_tmp = img + ".tmp"
> +        subprocess.check_call(["cp", "-f", os_img, img_tmp])
> +        subprocess.check_call(["qemu-img", "resize", img_tmp, "+50G"])
> +        ci_img = self._gen_cloud_init_iso()
> +
> +        self.boot(img_tmp, extra_args = ["-cdrom", ci_img])
> +        self.wait_ssh(wait_root=True)
> +        # Fix for slow ssh login.
> +        self.ssh_root("chmod -x /etc/update-motd.d/*")
> +        self.ssh_root("touch /etc/cloud/cloud-init.disabled")
> +        # Disable auto upgrades.
> +        # We want to keep the VM system state stable.
> +        self.ssh_root('sed -ie \'s/"1"/"0"/g\' 
> /etc/apt/apt.conf.d/20auto-upgrades')
> +
> +        # If the user chooses *not* to do the second phase,
> +        # then we will jump right to the graceful shutdown
> +        if self._config['install_cmds'] != "":
> +            # Don't check the status in case the guest hang up too quickly
> +            self.ssh_root("sync && reboot")
> +
> +            self.wait_ssh(wait_root=True)
> +            # The previous update sometimes doesn't survive a reboot, so do 
> it again
> +            self.ssh_root("sed -ie s/^#\ deb-src/deb-src/g 
> /etc/apt/sources.list")
> +
> +            # Issue the install commands.
> +            # This can be overriden by the user in the config .yml.
> +            install_cmds = self._config['install_cmds'].split(',')
> +            for cmd in install_cmds:
> +                self.ssh_root(cmd)
> +        self.graceful_shutdown()
> +        self.wait()
> +        os.rename(img_tmp, img)
> +        return 0
> +
> +if __name__ == "__main__":
> +    sys.exit(basevm.main(UbuntuAarch64VM, DEFAULT_CONFIG))


-- 
Alex Bennée