Re: [PATCH v4 00/10] Introduce new acpi/smbios avocado tests using biosbits

[Ani Sinha]

On Tue, Oct 18, 2022 at 9:01 PM Alex Bennée <alex.bennee@linaro.org> wrote:
>
>
> Ani Sinha <ani@anisinha.ca> writes:
>
> > Please see the README file in patch 9 for more details.
> > Sample runs are as follows:
> >
> > Passed tests:
> >
> > $ ./tests/venv/bin/avocado run -t acpi tests/avocado
> > Fetching asset from 
> > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
> > JOB ID     : e95c7413e996bfb59389839e5ca5105464ef098f
> > JOB LOG    : 
> > /home/anisinha/avocado/job-results/job-2022-10-14T19.15-e95c741/job.log
> >  (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS 
> > (33.08 s)
> > RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
> > CANCEL 0
> > JOB TIME   : 39.18 s
>
> Hmm something went wrong:
>
>   ➜  ./tests/venv/bin/avocado run -t acpi tests/avocado
>   Fetching asset from 
> tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
>   JOB ID     : 08b6d26bf52a22ccbe1908b14d5b7b27f76ad7fa
>   JOB LOG    : 
> /home/alex/avocado/job-results/job-2022-10-18T16.29-08b6d26/job.log
>    (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: 
> ERROR: Error while generating the bits iso. Pass V=1 in the environment to 
> get more details. Command '[
>   '/var/tmp/acpi-bits-yy4mfxrz.tmp/grub-inst-x86_64-efi/bin/grub-mkrescue', 
> '-o', '/var/tmp/acpi-bits-yy4mfxrz.tmp/bits-2020.iso', 
> '/var/tmp/acpi-bits-yy4mfxrz.t... (7.21 s)
>   RESULTS    : PASS 0 | ERROR 1 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
> CANCEL 0
>   JOB TIME   : 7.79 s
>
> have we added another binary dependency?

Try with V=1 , Maybe you do not have xorriso? If that is the case, you
need to install it in your test environment.

>
> >
> >
> > $ ./tests/venv/bin/avocado run ./tests/avocado/acpi-bits.py
> > Fetching asset from 
> > ./tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
> > JOB ID     : c6e588594bceb2c5fdbdf45095559c58e4030f9c
> > JOB LOG    : 
> > /home/anisinha/avocado/job-results/job-2022-10-14T19.13-c6e5885/job.log
> >  (1/1) ./tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: 
> > PASS (32.29 s)
> > RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
> > CANCEL 0
> > JOB TIME   : 39.27 s
> >
> > When PSS tests are enabled:
> >
> > $ git diff
> > diff --git a/tests/avocado/acpi-bits/bits-tests/testacpi.py2 
> > b/tests/avocado/acpi-bits/bits-tests/testacpi.py2
> > index f818a9cce6..18dc818d62 100644
> > --- a/tests/avocado/acpi-bits/bits-tests/testacpi.py2
> > +++ b/tests/avocado/acpi-bits/bits-tests/testacpi.py2
> > @@ -40,8 +40,8 @@ import time
> >
> >  def register_tests():
> >      testsuite.add_test("ACPI _MAT (Multiple APIC Table Entry) under 
> > Processor objects", test_mat, submenu="ACPI Tests")
> > -#    testsuite.add_test("ACPI _PSS (Pstate) table conformance tests", 
> > test_pss, submenu="ACPI Tests")
> > -#    testsuite.add_test("ACPI _PSS (Pstate) runtime tests", test_pstates, 
> > submenu="ACPI Tests")
> > +    testsuite.add_test("ACPI _PSS (Pstate) table conformance tests", 
> > test_pss, submenu="ACPI Tests")
> > +    testsuite.add_test("ACPI _PSS (Pstate) runtime tests", test_pstates, 
> > submenu="ACPI Tests")
> >      testsuite.add_test("ACPI DSDT (Differentiated System Description 
> > Table)", test_dsdt, submenu="ACPI Tests")
> >      testsuite.add_test("ACPI FACP (Fixed ACPI Description Table)", 
> > test_facp, submenu="ACPI Tests")
> >      testsuite.add_test("ACPI HPET (High Precision Event Timer Table)", 
> > test_hpet, submenu="ACPI Tests")
> >
> > $ ./tests/venv/bin/avocado run -t acpi tests/avocado
> > Fetching asset from 
> > tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
> > JOB ID     : 16a8b8c7f8dc87c4b84f8a2f95135bb63464a71e
> > JOB LOG    : 
> > /home/anisinha/avocado/job-results/job-2022-10-14T22.43-16a8b8c/job.log
> >  (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: FAIL: 
> > '1' != '0'\n- 1\n+ 0\n : Some bits tests seems to have failed. Please check 
> > the test logs for more info. (33.01 s)
> > RESULTS    : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
> > CANCEL 0
> > JOB TIME   : 39.17 s
> >
> >
> > Changelog:
> > v4:
> >   - renamed the bits test files as .py2.
> >   - fixed a bug with missing log file parsing. dump bits logs automatically
> >     for test failures (no need to re-run with V=1).
> >   - fixed issues with README file and enhanced it.
> >   - cosmetic comment updates.
> > v3:
> >   - converted the existing test to avocado tests as per the popular
> >     recommendation. Added appropriate tags.
> >   - artifact download URL modified based on gitlab-CI URL.
> >
> >   For biosbits repo:
> >   - moved to a gitlab repo with me being the maintainer.
> >   - added .gitlab-ci.yml file to generate the artifacts.
> > v2:
> >  - a new class of python based tests introduced that is separate from 
> > avocado
> >    tests or qtests. Can be run by using "make check-pytest".
> >  - acpi biosbits tests are the first tests to use pytest environment.
> >  - bios bits tests now download the bits binary archives from a remote
> >    repository if they are not found locally. The test skips if download
> >    fails.
> >  - A new environment variable is introduced that can be passed by the tester
> >    to specify the location of the bits archives locally. test skips if the
> >    bits binaries are not found in that location.
> >  - if pip install of python module fails for whatever reaoson, the test 
> > skips.
> >  - misc code fixes including spell check of the README doc. README has been
> >    updated as well.
> >  - addition of SPDX license headers to bits test files.
> >  - update MAINTAINERS to reflect the new pytest test class.
> >
> > For biosbits repo:
> >  - added Dockerfile and build script. Made bios bits build on gcc 11.
> >    https://github.com/ani-sinha/bits/blob/bits-qemu-logging/Dockerfile
> >    
> > https://github.com/ani-sinha/bits/blob/bits-qemu-logging/build-artifacts.sh
> >    The build script generates the zip archive and tarball used by the test.
> >
> > v1: initial patchset. uses qtest to implement the bios bits tests.
> >
> > Cc: Qemu Devel <qemu-devel@nongnu.org>
> > Cc: Daniel P. Berrangé <berrange@redhat.com>
> > Cc: Paolo Bonzini <pbonzini@redhat.com>
> > Cc: Maydell Peter <peter.maydell@linaro.org>
> > Cc: John Snow <jsnow@redhat.com>
> > Cc: Thomas Huth <thuth@redhat.com>
> > Cc: Alex Bennée <alex.bennee@linaro.org>
> > Cc: Igor Mammedov <imammedo@redhat.com>
> > Cc: Michael Tsirkin <mst@redhat.com>
> >
> >
> > Ani Sinha (10):
> >   acpi/tests/avocado/bits: initial commit of test scripts that are run
> >     by biosbits
> >   acpi/tests/avocado/bits: add SPDX license identifiers for bios bits
> >     tests
> >   acpi/tests/avocado/bits: disable acpi PSS tests that are failing in
> >     biosbits
> >   acpi/tests/avocado/bits: add smilatency test suite from bits in order
> >     to disable it
> >   acpi/tests/avocado/bits: add SPDX license identifiers for bios bits
> >     smilatency tests
> >   acpi/tests/avocado/bits: disable smilatency test since it does not
> >     pass everytime
> >   acpi/tests/avocado/bits: add biosbits config file for running bios
> >     tests
> >   acpi/tests/avocado/bits: add acpi and smbios avocado tests that uses
> >     biosbits
> >   acpi/tests/avocado/bits: add a README file to describe the test
> >   MAINTAINERS: add myself as the maintainer for acpi biosbits avocado
> >     tests
> >
> >  MAINTAINERS                                   |    6 +
> >  tests/avocado/acpi-bits.py                    |  353 +++
> >  tests/avocado/acpi-bits/README                |  128 +
> >  .../acpi-bits/bits-config/bits-cfg.txt        |   18 +
> >  tests/avocado/acpi-bits/bits-tests/smbios.py2 | 2434 +++++++++++++++++
> >  .../acpi-bits/bits-tests/smilatency.py2       |  107 +
> >  .../avocado/acpi-bits/bits-tests/testacpi.py2 |  287 ++
> >  .../acpi-bits/bits-tests/testcpuid.py2        |   87 +
> >  8 files changed, 3420 insertions(+)
> >  create mode 100644 tests/avocado/acpi-bits.py
> >  create mode 100644 tests/avocado/acpi-bits/README
> >  create mode 100644 tests/avocado/acpi-bits/bits-config/bits-cfg.txt
> >  create mode 100644 tests/avocado/acpi-bits/bits-tests/smbios.py2
> >  create mode 100644 tests/avocado/acpi-bits/bits-tests/smilatency.py2
> >  create mode 100644 tests/avocado/acpi-bits/bits-tests/testacpi.py2
> >  create mode 100644 tests/avocado/acpi-bits/bits-tests/testcpuid.py2
>
>
> --
> Alex Bennée