[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [PATCH v6 09/10] acpi/tests/avocado/bits/doc: add a doc file to desc
From: |
Alex Bennée |
Subject: |
Re: [PATCH v6 09/10] acpi/tests/avocado/bits/doc: add a doc file to describe the acpi bits test |
Date: |
Thu, 20 Oct 2022 18:08:33 +0100 |
User-agent: |
mu4e 1.9.1; emacs 28.2.50 |
Ani Sinha <ani@anisinha.ca> writes:
> A doc file is added under docs/devel that describes the purpose of the various
> test files and gives guidance to developers on where and how to make changes.
>
> Cc: Daniel P. Berrange" <berrange@redhat.com>
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Maydell Peter <peter.maydell@linaro.org>
> Cc: John Snow <jsnow@redhat.com>
> Cc: Thomas Huth <thuth@redhat.com>
> Cc: Alex Bennée <alex.bennee@linaro.org>
> Cc: Igor Mammedov <imammedo@redhat.com>
> Cc: Michael Tsirkin <mst@redhat.com>
> Signed-off-by: Ani Sinha <ani@anisinha.ca>
> ---
> docs/devel/acpi-bits.rst | 148 +++++++++++++++++++++++++++++++++++++++
> 1 file changed, 148 insertions(+)
> create mode 100644 docs/devel/acpi-bits.rst
You will also need to add it to the index-build.rst:
testing
+ acpi-bits
qtest
Other wise the build will complains.
>
> diff --git a/docs/devel/acpi-bits.rst b/docs/devel/acpi-bits.rst
> new file mode 100644
> index 0000000000..06d50b76d9
> --- /dev/null
> +++ b/docs/devel/acpi-bits.rst
> @@ -0,0 +1,148 @@
> +=============================================================================
> +ACPI/SMBIOS AVOCADO TESTS USING BIOSBITS
No need to use all caps here.
> +=============================================================================
> +
> +Biosbits is a software written by Josh Triplett that can be downloaded
> +from https://biosbits.org/. The github codebase can be found here:
> +https://github.com/biosbits/bits/tree/master.
`here <https://github.com/biosbits/bits/tree/master>`__.
> It is a software that executes
> +the bios components such as acpi and smbios tables directly through acpica
> +bios interpreter (a freely available C based library written by Intel,
> +downloadable from https://acpica.org/ and is included with biosbits) without
> an
> +operating system getting involved in between.
> +There are several advantages to directly testing the bios in a real physical
> +machine or VM as opposed to indirectly discovering bios issues through the
> +operating system. For one thing, the OSes tend to hide bios problems from the
> +end user. The other is that we have more control of what we wanted to test
> +and how by directly using acpica interpreter on top of the bios on a running
> +system. More details on the inspiration for developing biosbits and its real
> +life uses can be found in (a) and (b).
[#a]_ and [#b]_ for proper footnotes
> +This directory contains tests written in python using avocado framework that
> +exercizes the QEMU bios components using biosbits and reports test
> failures.
exercises
> +For QEMU, we maintain a fork of bios bits in gitlab along with all the
> +dependent submodules:
> +https://gitlab.com/qemu-project/biosbits-bits
> +This fork contains numerous fixes, a newer acpica and changes specific to
> +running this avocado QEMU tests using bits. The author of this document
> +is the sole maintainer of the QEMU fork of bios bits repo.
> +
> +Under the directory ``tests/avocado/``, ``acpi-bits.py`` is a QEMU avocado
> +test that drives all this.
> +
> +A brief description of the various test files follows.
> +
> +Under ``tests/avocado/`` as the root we have:
> +
> +::
> +
> + ├── acpi-bits
> + │ ├── bits-config
> + │ │ └── bits-cfg.txt
> + │ ├── bits-tests
> + │ │ ├── smbios.py2
> + │ │ ├── smilatency.py2
> + │ │ ├── testacpi.py2
> + │ │ └── testcpuid.py2
> + │ └── README
> + ├── acpi-bits.py
> +
> +* ``tests/avocado``:
> +
> + ``acpi-bits.py``:
> + This is the main python avocado test script that generates a
> + biosbits iso. It then spawns a QEMU VM with it, collects the log and
> reports
> + test failures. This is the script one would be interested in if they
> wanted
> + to add or change some component of the log parsing, add a new command line
> + to alter how QEMU is spawned etc. Test writers typically would not need to
> + modify this script unless they wanted to enhance or change the log parsing
> + for their tests. Following environment variables are used in this test:
> +
> + **V=1** : This enables verbose mode for the test. It dumps the entire log
> + from bios bits and also more details in case failure happens. It is
> + useful for debugging the test failures or tests themselves.
As there is only one environment variable you could just describe it
inline, e.g.:
Test writers typically would not need to modify this script unless
they wanted to enhance or change the log parsing for their tests. To
enable debugging you can set the ``V=1`` environment variable. This
will cause the test to dump the entire log from bios bits and also
more details in case failure happens.
> +
> + In order to run this test, please perform the following steps from the
> QEMU
> + build directory:
> + ::
> +
> + $ make check-venv (needed only the first time to create the venv)
> + $ ./tests/venv/bin/avocado run -t acpi tests/avocado
> +
> + The above will run all acpi avocado tests including this one.
> + In order to run the individual tests, perform the following:
> + ::
> +
> + $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py --tap -
> +
> + The above will produce output in tap format. You can omit "--tap -" in the
> + end and it will produce output like the following:
> + ::
> +
> + $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py
> + Fetching asset from
> tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
> + JOB ID : eab225724da7b64c012c65705dc2fa14ab1defef
> + JOB LOG :
> /home/anisinha/avocado/job-results/job-2022-10-10T17.58-eab2257/job.log
> + (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits:
> PASS (33.09 s)
> + RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0
> | CANCEL 0
> + JOB TIME : 39.22 s
> +
> + You can inspect the log file for more information about the run or in
> order
> + to diagnoze issues. If you pass V=1 in the environment, more diagnostic
> logs
> + would be found in the test log.
> +
> +* ``tests/avocado/acpi-bits/bits-config``:
> +
> + This location contains biosbits configuration files that determine how the
> + software runs the tests.
> +
> + ``bits-config.txt``:
> + This is the biosbits config file that determines what tests
> + or actions are performed by bits. The description of the config options
> are
> + provided in the file itself.
> +
> +* ``tests/avocado/acpi-bits/bits-tests``:
> +
> + This directory contains biosbits python based tests that are run from
> within
> + the biosbits environment in the spawned VM. New additions of test cases
> can
> + be made in the appropriate test file. For example, new acpi tests can go
> + into testacpi.py2 and one would call testsuite.add_test() to register the
> new
> + test so that it gets executed as a part of the ACPI tests.
> + It might be occasionally necessary to disable some subtests or add a new
> + test that belongs to a test suite not already present in this directory.
> To
> + do this, please clone the bits source from
> + https://gitlab.com/qemu-project/biosbits-bits/-/tree/qemu-bits.
> + Note that this is the "qemu-bits" branch and not the "bits" branch of the
> + repository. "qemu-bits" is the branch where we have made all the QEMU
> + specific enhancements and we must use the source from this branch only.
> + Copy the test suite/script that needs modification (addition of new tests
> + or disabling them) from python directory into this directory. For
> + example, in order to change cpuid related tests, copy the following
> + file into this directory and rename it with .py2 extension:
> +
> https://gitlab.com/qemu-project/biosbits-bits/-/blob/qemu-bits/python/testcpuid.py
> + Then make your additions and changes here. Therefore, the steps are:
> +
> + (a) Copy unmodified test script to this directory from bits source.
> + (b) Add a SPDX license header.
> + (c) Perform modifications to the test.
Maybe unordered lists would be better here
> +
> + Commits (a), (b) and (c) should go under separate commits so that the
> original
> + test script and the changes we have made are separated and clear.
> +
> + The test framework will then use your modified test script to run the
> test.
> + No further changes would be needed. Please check the logs to make sure
> that
> + appropriate changes have taken effect.
> +
> + The tests have an extension .py2 in order to indicate that:
> +
> + (a) They are python2.7 based scripts and not python 3 scripts.
> + (b) They are run from within the bios bits VM and is not subjected to QEMU
> + build/test python script maintainance and dependency resolutions.
> + (c) They need not be loaded by avocado framework when running tests.
> +
> +
> +Author: Ani Sinha <ani@anisinha.ca>
> +
> +References:
> +-----------
> +(a)
> https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
> +(b) https://www.youtube.com/watch?v=36QIepyUuhg
> +
and then:
.. [#a]
https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
.. [#b] https://www.youtube.com/watch?v=36QIepyUuhg
--
Alex Bennée
- Re: [PATCH v6 01/10] acpi/tests/avocado/bits: initial commit of test scripts that are run by biosbits, (continued)
- [PATCH v6 08/10] acpi/tests/avocado/bits: add acpi and smbios avocado tests that uses biosbits, Ani Sinha, 2022/10/20
- [PATCH v6 06/10] acpi/tests/avocado/bits: disable smilatency test since it does not pass everytime, Ani Sinha, 2022/10/20
- [PATCH v6 04/10] acpi/tests/avocado/bits: add smilatency test suite from bits in order to disable it, Ani Sinha, 2022/10/20
- [PATCH v6 07/10] acpi/tests/avocado/bits: add biosbits config file for running bios tests, Ani Sinha, 2022/10/20
- [PATCH v6 09/10] acpi/tests/avocado/bits/doc: add a doc file to describe the acpi bits test, Ani Sinha, 2022/10/20
- Re: [PATCH v6 09/10] acpi/tests/avocado/bits/doc: add a doc file to describe the acpi bits test,
Alex Bennée <=
- [PATCH v6 02/10] acpi/tests/avocado/bits: add SPDX license identifiers for bios bits tests, Ani Sinha, 2022/10/20
- [PATCH v6 10/10] MAINTAINERS: add myself as the maintainer for acpi biosbits avocado tests, Ani Sinha, 2022/10/20
- Re: [PATCH v6 00/10] Introduce new acpi/smbios avocado tests using biosbits, Michael S. Tsirkin, 2022/10/20
- Re: [PATCH v6 00/10] Introduce new acpi/smbios avocado tests using biosbits, Ani Sinha, 2022/10/20
- Re: [PATCH v6 00/10] Introduce new acpi/smbios avocado tests using biosbits, Michael S. Tsirkin, 2022/10/20
- Re: [PATCH v6 00/10] Introduce new acpi/smbios avocado tests using biosbits, Ani Sinha, 2022/10/20
- Re: [PATCH v6 00/10] Introduce new acpi/smbios avocado tests using biosbits, Michael S. Tsirkin, 2022/10/20
- Re: [PATCH v6 00/10] Introduce new acpi/smbios avocado tests using biosbits, Ani Sinha, 2022/10/20
- Re: [PATCH v6 00/10] Introduce new acpi/smbios avocado tests using biosbits, Ani Sinha, 2022/10/20
- Re: [PATCH v6 00/10] Introduce new acpi/smbios avocado tests using biosbits, Michael S. Tsirkin, 2022/10/21