[PULL 00/42] Introduce new functional test framework

[Thomas Huth]

 Hi!

The following changes since commit e638d685ec2a0700fb9529cbd1b2823ac4120c53:

  Open 9.2 development tree (2024-09-03 09:18:43 -0700)

are available in the Git repository at:

  https://gitlab.com/thuth/qemu.git tags/pull-request-2024-09-04

for you to fetch changes up to c3e24cff2b27d63ac4b56ac6d38ef1ae3a27d92f:

  docs/devel/testing: Add documentation for functional tests (2024-09-04 
12:28:00 +0200)

----------------------------------------------------------------
* Bump Avocado to version 103
* Introduce new functional test framework for Python-based tests
* Convert many Avocado tests to the new functional test framework

----------------------------------------------------------------
Cleber Rosa (4):
      tests/avocado: machine aarch64: standardize location and RO access
      tests/avocado/boot_xen.py: fetch kernel during test setUp()
      tests/avocado/machine_aarch64_sbsaref.py: allow for rw usage of image
      Bump avocado to 103.0

Daniel P. Berrangé (2):
      tests/functional: add a module for handling asset download & caching
      tests/functional: enable pre-emptive caching of assets

Philippe Mathieu-Daudé (7):
      tests/functional: Convert ARM Integrator/CP avocado tests
      tests/functional: Convert Aarch64 SBSA-Ref avocado tests
      tests/functional: Convert Aarch64 Virt machine avocado tests
      tests/functional: Convert mips64el Fuloong2e avocado test (1/2)
      tests/functional: Add QemuUserTest class
      tests/functional: Convert ARM bFLT linux-user avocado test
      tests/avocado: Remove unused QemuUserTest class

Thomas Huth (29):
      tests/avocado/avocado_qemu: Fix the "from" statements in linuxtest.py
      tests/avocado/boot_linux_console: Remove the s390x subtest
      python: Install pycotap in our venv if necessary
      tests/functional: Add base classes for the upcoming pytest-based tests
      tests/functional: Set up logging
      tests/Makefile.include: Increase the level of indentation in the help text
      tests/functional: Prepare the meson build system for the functional tests
      tests/functional: Convert simple avocado tests into standalone python 
tests
      tests/functional: Convert avocado tests that just need a small adjustment
      tests/functional: Allow asset downloading with concurrent threads
      tests/functional: Convert some tests that download files via fetch_asset()
      tests/functional: Add a function for extracting files from an archive
      tests/functional: Convert some avocado tests that needed 
avocado.utils.archive
      tests/functional: Convert the s390x avocado tests into standalone tests
      tests/functional: Convert the x86_cpu_model_versions test
      tests/functional: Convert the microblaze avocado tests into standalone 
tests
      tests/functional: Convert the virtio_gpu avocado test into a standalone 
test
      tests/functional: Convert most ppc avocado tests into standalone tests
      tests/functional: Convert the ppc_amiga avocado test into a standalone 
test
      tests/functional: Convert the ppc_hv avocado test into a standalone test
      tests/functional: Convert the m68k nextcube test with tesseract
      tests/functional: Convert the acpi-bits test into a standalone test
      tests/functional: Convert the rx_gdbsim avocado test into a standalone 
test
      tests/functional: Convert the linux_initrd avocado test into a standalone 
test
      gitlab-ci: Add "check-functional" to the build tests
      docs/devel: Split testing docs from the build docs and move to separate 
folder
      docs/devel/testing: Split the Avocado documentation into a separate file
      docs/devel/testing: Rename avocado_qemu.Test class
      docs/devel/testing: Add documentation for functional tests

 MAINTAINERS                                        |  63 ++-
 docs/devel/index-build.rst                         |  14 +-
 docs/devel/index.rst                               |   1 +
 docs/devel/{ => testing}/acpi-bits.rst             |  62 +--
 docs/devel/testing/avocado.rst                     | 581 +++++++++++++++++++++
 docs/devel/{ => testing}/ci-definitions.rst.inc    |   0
 docs/devel/{ => testing}/ci-jobs.rst.inc           |   0
 docs/devel/{ => testing}/ci-runners.rst.inc        |   0
 docs/devel/{ => testing}/ci.rst                    |   0
 docs/devel/testing/functional.rst                  | 338 ++++++++++++
 docs/devel/{ => testing}/fuzzing.rst               |   0
 docs/devel/testing/index.rst                       |  16 +
 docs/devel/{testing.rst => testing/main.rst}       | 577 +-------------------
 docs/devel/{ => testing}/qgraph.rst                |   0
 docs/devel/{ => testing}/qtest.rst                 |   0
 .gitlab-ci.d/buildtest-template.yml                |  14 +-
 .gitlab-ci.d/buildtest.yml                         |  60 +--
 python/wheels/pycotap-1.3.1-py3-none-any.whl       | Bin 0 -> 5119 bytes
 pythondeps.toml                                    |   3 +-
 tests/Makefile.include                             |  44 +-
 tests/avocado/avocado_qemu/__init__.py             |  17 -
 tests/avocado/avocado_qemu/linuxtest.py            |   4 +-
 tests/avocado/boot_linux_console.py                |  20 -
 tests/avocado/boot_xen.py                          |  13 +-
 tests/avocado/load_bflt.py                         |  54 --
 tests/avocado/machine_arm_n8x0.py                  |   0
 tests/avocado/machine_microblaze.py                |  61 ---
 tests/avocado/machine_mips_loongson3v.py           |  39 --
 tests/avocado/machine_sparc64_sun4u.py             |  36 --
 tests/avocado/ppc_amiga.py                         |  38 --
 tests/avocado/tesseract_utils.py                   |  46 --
 .../acpi-bits/bits-config/bits-cfg.txt             |   0
 .../acpi-bits/bits-tests/smbios.py2                |   0
 .../acpi-bits/bits-tests/smilatency.py2            |   0
 .../acpi-bits/bits-tests/testacpi.py2              |   0
 .../acpi-bits/bits-tests/testcpuid.py2             |   0
 tests/functional/meson.build                       | 205 ++++++++
 tests/functional/qemu_test/__init__.py             |  14 +
 tests/functional/qemu_test/asset.py                | 171 ++++++
 tests/functional/qemu_test/cmd.py                  | 193 +++++++
 tests/functional/qemu_test/config.py               |  36 ++
 tests/functional/qemu_test/tesseract.py            |  35 ++
 tests/functional/qemu_test/testcase.py             | 202 +++++++
 tests/functional/qemu_test/utils.py                |  56 ++
 .../test_aarch64_sbsaref.py}                       | 148 ++----
 .../test_aarch64_virt.py}                          |  71 +--
 .../acpi-bits.py => functional/test_acpi_bits.py}  |  83 +--
 tests/functional/test_arm_bflt.py                  |  44 ++
 .../test_arm_canona1100.py}                        |  30 +-
 .../test_arm_integratorcp.py}                      |  56 +-
 .../test_avr_mega2560.py}                          |  22 +-
 .../test_cpu_queries.py}                           |  12 +-
 .../test_empty_cpu_model.py}                       |   7 +-
 .../test_info_usernet.py}                          |  11 +-
 .../test_linux_initrd.py}                          |  37 +-
 .../test_loongarch64_virt.py}                      |  46 +-
 .../test_m68k_nextcube.py}                         |  29 +-
 .../test_mem_addr_space.py}                        |  52 +-
 tests/functional/test_microblaze_s3adsp1800.py     |  40 ++
 tests/functional/test_microblazeel_s3adsp1800.py   |  42 ++
 .../test_mips64el_fuloong2e.py}                    |  33 +-
 tests/functional/test_mips64el_loongson3v.py       |  39 ++
 .../test_netdev_ethtool.py}                        |  51 +-
 .../test_pc_cpu_hotplug_props.py}                  |  11 +-
 .../test_ppc64_hv.py}                              |  52 +-
 .../test_ppc64_powernv.py}                         |  56 +-
 .../test_ppc64_pseries.py}                         |  56 +-
 .../ppc_405.py => functional/test_ppc_405.py}      |  29 +-
 .../ppc_prep_40p.py => functional/test_ppc_40p.py} |  65 +--
 .../ppc_74xx.py => functional/test_ppc_74xx.py}    |  74 ++-
 tests/functional/test_ppc_amiga.py                 |  43 ++
 .../test_ppc_bamboo.py}                            |  33 +-
 .../test_ppc_mpc8544ds.py}                         |  29 +-
 .../test_ppc_virtex_ml507.py}                      |  29 +-
 .../test_rx_gdbsim.py}                             |  54 +-
 .../test_s390x_ccw_virtio.py}                      |  81 ++-
 .../test_s390x_topology.py}                        |  86 ++-
 tests/functional/test_sparc64_sun4u.py             |  41 ++
 .../version.py => functional/test_version.py}      |  13 +-
 .../test_virtio_gpu.py}                            |  64 +--
 .../test_virtio_version.py}                        |   8 +-
 .../test_x86_cpu_model_versions.py}                |  65 +--
 tests/meson.build                                  |   1 +
 83 files changed, 2915 insertions(+), 1841 deletions(-)
 rename docs/devel/{ => testing}/acpi-bits.rst (78%)
 create mode 100644 docs/devel/testing/avocado.rst
 rename docs/devel/{ => testing}/ci-definitions.rst.inc (100%)
 rename docs/devel/{ => testing}/ci-jobs.rst.inc (100%)
 rename docs/devel/{ => testing}/ci-runners.rst.inc (100%)
 rename docs/devel/{ => testing}/ci.rst (100%)
 create mode 100644 docs/devel/testing/functional.rst
 rename docs/devel/{ => testing}/fuzzing.rst (100%)
 create mode 100644 docs/devel/testing/index.rst
 rename docs/devel/{testing.rst => testing/main.rst} (67%)
 rename docs/devel/{ => testing}/qgraph.rst (100%)
 rename docs/devel/{ => testing}/qtest.rst (100%)
 create mode 100644 python/wheels/pycotap-1.3.1-py3-none-any.whl
 delete mode 100644 tests/avocado/load_bflt.py
 mode change 100644 => 100755 tests/avocado/machine_arm_n8x0.py
 delete mode 100644 tests/avocado/machine_microblaze.py
 delete mode 100644 tests/avocado/machine_mips_loongson3v.py
 delete mode 100644 tests/avocado/machine_sparc64_sun4u.py
 delete mode 100644 tests/avocado/ppc_amiga.py
 delete mode 100644 tests/avocado/tesseract_utils.py
 rename tests/{avocado => functional}/acpi-bits/bits-config/bits-cfg.txt (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/smbios.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/smilatency.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/testacpi.py2 (100%)
 rename tests/{avocado => functional}/acpi-bits/bits-tests/testcpuid.py2 (100%)
 create mode 100644 tests/functional/meson.build
 create mode 100644 tests/functional/qemu_test/__init__.py
 create mode 100644 tests/functional/qemu_test/asset.py
 create mode 100644 tests/functional/qemu_test/cmd.py
 create mode 100644 tests/functional/qemu_test/config.py
 create mode 100644 tests/functional/qemu_test/tesseract.py
 create mode 100644 tests/functional/qemu_test/testcase.py
 create mode 100644 tests/functional/qemu_test/utils.py
 rename tests/{avocado/machine_aarch64_sbsaref.py => 
functional/test_aarch64_sbsaref.py} (55%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_aarch64_virt.py => 
functional/test_aarch64_virt.py} (68%)
 mode change 100644 => 100755
 rename tests/{avocado/acpi-bits.py => functional/test_acpi_bits.py} (86%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_arm_bflt.py
 rename tests/{avocado/machine_arm_canona1100.py => 
functional/test_arm_canona1100.py} (51%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_arm_integratorcp.py => 
functional/test_arm_integratorcp.py} (63%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py} (72%)
 mode change 100644 => 100755
 rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (85%)
 mode change 100644 => 100755
 rename tests/{avocado/empty_cpu_model.py => 
functional/test_empty_cpu_model.py} (84%)
 mode change 100644 => 100755
 rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%)
 mode change 100644 => 100755
 rename tests/{avocado/linux_initrd.py => functional/test_linux_initrd.py} (76%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_loongarch.py => 
functional/test_loongarch64_virt.py} (54%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_m68k_nextcube.py => 
functional/test_m68k_nextcube.py} (75%)
 mode change 100644 => 100755
 rename tests/{avocado/mem-addr-space-check.py => 
functional/test_mem_addr_space.py} (93%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_microblaze_s3adsp1800.py
 create mode 100755 tests/functional/test_microblazeel_s3adsp1800.py
 rename tests/{avocado/machine_mips_fuloong2e.py => 
functional/test_mips64el_fuloong2e.py} (58%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_mips64el_loongson3v.py
 rename tests/{avocado/netdev-ethtool.py => functional/test_netdev_ethtool.py} 
(66%)
 mode change 100644 => 100755
 rename tests/{avocado/pc_cpu_hotplug_props.py => 
functional/test_pc_cpu_hotplug_props.py} (90%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_hv_tests.py => functional/test_ppc64_hv.py} (85%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_powernv.py => functional/test_ppc64_powernv.py} (71%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_pseries.py => functional/test_ppc64_pseries.py} (76%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (53%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_prep_40p.py => functional/test_ppc_40p.py} (51%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_ppc_amiga.py
 rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py} (60%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_mpc8544ds.py => functional/test_ppc_mpc8544ds.py} 
(55%)
 mode change 100644 => 100755
 rename tests/{avocado/ppc_virtex_ml507.py => 
functional/test_ppc_virtex_ml507.py} (60%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_rx_gdbsim.py => functional/test_rx_gdbsim.py} 
(58%)
 mode change 100644 => 100755
 rename tests/{avocado/machine_s390_ccw_virtio.py => 
functional/test_s390x_ccw_virtio.py} (85%)
 mode change 100644 => 100755
 rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} 
(88%)
 mode change 100644 => 100755
 create mode 100755 tests/functional/test_sparc64_sun4u.py
 rename tests/{avocado/version.py => functional/test_version.py} (78%)
 mode change 100644 => 100755
 rename tests/{avocado/virtio-gpu.py => functional/test_virtio_gpu.py} (73%)
 mode change 100644 => 100755
 rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} 
(98%)
 mode change 100644 => 100755
 rename tests/{avocado/x86_cpu_model_versions.py => 
functional/test_x86_cpu_model_versions.py} (92%)
 mode change 100644 => 100755