qemu-commits
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Qemu-commits] [qemu/qemu] 2ca6e2: Python: expose QEMUMachine's temporar


From: Peter Maydell
Subject: [Qemu-commits] [qemu/qemu] 2ca6e2: Python: expose QEMUMachine's temporary directory
Date: Wed, 02 Jun 2021 09:08:33 -0700

  Branch: refs/heads/master
  Home:   https://github.com/qemu/qemu
  Commit: 2ca6e26cea73fa1d270f73392e8b87f3e67e6a2b
      
https://github.com/qemu/qemu/commit/2ca6e26cea73fa1d270f73392e8b87f3e67e6a2b
  Author: Cleber Rosa <crosa@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M python/qemu/machine.py
    M python/qemu/qtest.py
    M tests/qemu-iotests/iotests.py

  Log Message:
  -----------
  Python: expose QEMUMachine's temporary directory

Each instance of qemu.machine.QEMUMachine currently has a "test
directory", which may not have any relation to a "test", and it's
really a temporary directory.

Users instantiating the QEMUMachine class will be able to set the
location of the directory that will *contain* the QEMUMachine unique
temporary directory, so that parameter name has been changed from
test_dir to base_temp_dir.

A property has been added to allow users to access it without using
private attributes, and with that, the directory is created on first
use of the property.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20210211220146.2525771-3-crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: f084e148aa401f71c33da4268f73815e0dc2b262
      
https://github.com/qemu/qemu/commit/f084e148aa401f71c33da4268f73815e0dc2b262
  Author: Cleber Rosa <crosa@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M tests/acceptance/virtiofs_submounts.py

  Log Message:
  -----------
  tests/acceptance/virtiofs_submounts.py: add missing accel tag

The tag is useful to select tests that depend/use a particular
feature.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Message-Id: <20210412044644.55083-2-crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: c028691e65b9f45e7a8ff8ffbfb9a3818478b572
      
https://github.com/qemu/qemu/commit/c028691e65b9f45e7a8ff8ffbfb9a3818478b572
  Author: Cleber Rosa <crosa@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M tests/acceptance/virtiofs_submounts.py

  Log Message:
  -----------
  tests/acceptance/virtiofs_submounts.py: evaluate string not length

If the vmlinuz variable is set to anything that evaluates to True,
then the respective arguments should be set.  If the variable contains
an empty string, than it will evaluate to False, and the extra
arguments will not be set.

This keeps the same logic, but improves readability a bit.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210412044644.55083-3-crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 976218cbe792c37c1af7840ca5113e37b5a51d95
      
https://github.com/qemu/qemu/commit/976218cbe792c37c1af7840ca5113e37b5a51d95
  Author: Cleber Rosa <crosa@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    A python/qemu/utils.py
    A tests/acceptance/info_usernet.py
    M tests/acceptance/linux_ssh_mips_malta.py
    M tests/acceptance/virtiofs_submounts.py
    M tests/vm/basevm.py

  Log Message:
  -----------
  Python: add utility function for retrieving port redirection

Slightly different versions for the same utility code are currently
present on different locations.  This unifies them all, giving
preference to the version from virtiofs_submounts.py, because of the
last tweaks added to it.

While at it, this adds a "qemu.utils" module to host the utility
function and a test.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210412044644.55083-4-crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
[Squashed in below fix. --js]
Signed-off-by: John Snow <jsnow@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20210601154546.130870-2-crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 7edee7ad9408696b9b8d40b5842a07a0c4e9b7a2
      
https://github.com/qemu/qemu/commit/7edee7ad9408696b9b8d40b5842a07a0c4e9b7a2
  Author: Cleber Rosa <crosa@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M tests/acceptance/avocado_qemu/__init__.py
    M tests/acceptance/linux_ssh_mips_malta.py
    M tests/acceptance/virtiofs_submounts.py

  Log Message:
  -----------
  Acceptance Tests: move useful ssh methods to base class

Both the virtiofs submounts and the linux ssh mips malta tests
contains useful methods related to ssh that deserve to be made
available to other tests.  Let's move them to an auxiliary, mix-in
class that will be used on the base LinuxTest class.

The method that helps with setting up an ssh connection will now
support both key and password based authentication, defaulting to key
based.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20210412044644.55083-5-crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 54914114aff5008b58d3cf01bf9e2274144875ca
      
https://github.com/qemu/qemu/commit/54914114aff5008b58d3cf01bf9e2274144875ca
  Author: Cleber Rosa <crosa@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M tests/acceptance/avocado_qemu/__init__.py
    M tests/acceptance/virtiofs_submounts.py

  Log Message:
  -----------
  Acceptance Tests: add port redirection for ssh by default

For users of the LinuxTest class, let's set up the VM with the port
redirection for SSH, instead of requiring each test to set the same
arguments.

It also sets the network device, by default, to virtio-net.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210412044644.55083-6-crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: d8c6a89968906af24ab27acd936013d3f937fc16
      
https://github.com/qemu/qemu/commit/d8c6a89968906af24ab27acd936013d3f937fc16
  Author: Cleber Rosa <crosa@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M tests/acceptance/avocado_qemu/__init__.py

  Log Message:
  -----------
  Acceptance Tests: make username/password configurable

This makes the username/password used for authentication configurable,
because some guest operating systems may have restrictions on accounts
to be used for logins, and it just makes it better documented.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210412044644.55083-7-crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: c6620c443d076bc0c80357e41f8f8d7fcdade6df
      
https://github.com/qemu/qemu/commit/c6620c443d076bc0c80357e41f8f8d7fcdade6df
  Author: Cleber Rosa <crosa@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M tests/acceptance/avocado_qemu/__init__.py
    M tests/acceptance/boot_linux.py
    M tests/acceptance/virtiofs_submounts.py

  Log Message:
  -----------
  Acceptance Tests: set up SSH connection by default after boot for LinuxTest

The LinuxTest specifically targets users that need to interact with Linux
guests.  So, it makes sense to give a connection by default, and avoid
requiring it as boiler-plate code.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210412044644.55083-8-crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: a273387aec43d2f2cff19b232c8c3e569a669971
      
https://github.com/qemu/qemu/commit/a273387aec43d2f2cff19b232c8c3e569a669971
  Author: Cleber Rosa <crosa@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M tests/acceptance/virtiofs_submounts.py

  Log Message:
  -----------
  tests/acceptance/virtiofs_submounts.py: remove launch_vm()

The LinuxTest class' launch_and_wait() method now behaves the same way
as this test's custom launch_vm(), so let's just use the upper layer
(common) method.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210412044644.55083-9-crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 1e4e7efa01f021e7abeb0c12ff7bb3758da22134
      
https://github.com/qemu/qemu/commit/1e4e7efa01f021e7abeb0c12ff7bb3758da22134
  Author: Cleber Rosa <crosa@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M docs/devel/testing.rst

  Log Message:
  -----------
  Acceptance Tests: add basic documentation on LinuxTest base class

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210412044644.55083-10-crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: fd1ce58d901bbe982db8c19ca6e1a63b30643150
      
https://github.com/qemu/qemu/commit/fd1ce58d901bbe982db8c19ca6e1a63b30643150
  Author: Cleber Rosa <crosa@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    A tests/acceptance/hotplug_cpu.py

  Log Message:
  -----------
  Acceptance Tests: introduce CPU hotplug test

Even though there are qtest based tests for hotplugging CPUs (from
which this test took some inspiration from), this one adds checks
from a Linux guest point of view.

It should also serve as an example for tests that follow a similar
pattern and need to interact with QEMU (via qmp) and with the Linux
guest via SSH.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Message-Id: <20210412044644.55083-11-crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: d214740c994f51370112ceda33a9d5546ff21c84
      
https://github.com/qemu/qemu/commit/d214740c994f51370112ceda33a9d5546ff21c84
  Author: Cleber Rosa <crosa@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M tests/acceptance/virtiofs_submounts.py

  Log Message:
  -----------
  tests/acceptance/virtiofs_submounts.py: fix setup of SSH pubkey

The public key argument should be a path to a file, and not the
public key data.

Reported-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20210412044644.55083-12-crosa@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 41787552de447733debe0616b716a0aa138242c7
      
https://github.com/qemu/qemu/commit/41787552de447733debe0616b716a0aa138242c7
  Author: Willian Rampazzo <willianr@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M tests/requirements.txt

  Log Message:
  -----------
  acceptance tests: bump Avocado version to 88.1

Besides some internal changes, new features, and bug fixes, on the QEMU side,
this version fixes the following message seen when running the acceptance
tests: "Error running method "pre_tests" of plugin "fetchasset": 'bytes'
object has no attribute 'encode'".

The release notes are available at
https://avocado-framework.readthedocs.io/en/latest/releases/88_0.html.

Signed-off-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210520204747.210764-2-willianr@redhat.com>
Acked-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: ee1a27235b7965bc5514555eec898f4d067fced2
      
https://github.com/qemu/qemu/commit/ee1a27235b7965bc5514555eec898f4d067fced2
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M python/qemu/console_socket.py

  Log Message:
  -----------
  python/console_socket: avoid one-letter variable

Fixes pylint warnings.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210527211715.394144-2-jsnow@redhat.com
Message-id: 20210517184808.3562549-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 07b71233a7ea77c0ec3687c3a3451865b3b899d3
      
https://github.com/qemu/qemu/commit/07b71233a7ea77c0ec3687c3a3451865b3b899d3
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M python/qemu/machine.py

  Log Message:
  -----------
  python/machine: use subprocess.DEVNULL instead of open(os.path.devnull)

One less file resource to manage, and it helps quiet some pylint >=
2.8.0 warnings about not using a with-context manager for the open call.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-3-jsnow@redhat.com
Message-id: 20210517184808.3562549-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 14b41797d5eb20fb9c6d0a1fe809e8422938f230
      
https://github.com/qemu/qemu/commit/14b41797d5eb20fb9c6d0a1fe809e8422938f230
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M python/qemu/machine.py

  Log Message:
  -----------
  python/machine: use subprocess.run instead of subprocess.Popen

use run() instead of Popen() -- to assert to pylint that we are not
forgetting to close a long-running program.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-4-jsnow@redhat.com
Message-id: 20210517184808.3562549-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 8825fed82a1949ed74f103c2ff26c4d71d2e4845
      
https://github.com/qemu/qemu/commit/8825fed82a1949ed74f103c2ff26c4d71d2e4845
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M python/qemu/console_socket.py

  Log Message:
  -----------
  python/console_socket: Add a pylint ignore

We manage cleaning up this resource ourselves. Pylint should shush.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-5-jsnow@redhat.com
Message-id: 20210517184808.3562549-5-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 63c33f3c286efe4c6474b53ae97915c9d1a6923a
      
https://github.com/qemu/qemu/commit/63c33f3c286efe4c6474b53ae97915c9d1a6923a
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M python/qemu/machine.py

  Log Message:
  -----------
  python/machine: Disable pylint warning for open() in _pre_launch

Shift the open() call later so that the pylint pragma applies *only* to
that one open() call. Add a note that suggests why this is safe: the
resource is unconditionally cleaned up in _post_shutdown().

_post_shutdown is called after failed launches (see launch()), and
unconditionally after every call to shutdown(), and therefore also on
__exit__.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-6-jsnow@redhat.com
Message-id: 20210517184808.3562549-6-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: a0eae17a59fcbcdc96af2ea2a6767d758ff4a916
      
https://github.com/qemu/qemu/commit/a0eae17a59fcbcdc96af2ea2a6767d758ff4a916
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M python/qemu/machine.py

  Log Message:
  -----------
  python/machine: disable warning for Popen in _launch()

We handle this resource rather meticulously in
shutdown/kill/wait/__exit__ et al, through the laborious mechanisms in
_do_shutdown().

Quiet this pylint warning here.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-7-jsnow@redhat.com
Message-id: 20210517184808.3562549-7-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 859aeb67d7372e63bd7bb2c7d063c2a49f2507ab
      
https://github.com/qemu/qemu/commit/859aeb67d7372e63bd7bb2c7d063c2a49f2507ab
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M python/qemu/machine.py

  Log Message:
  -----------
  python/machine: Trim line length to below 80 chars

One more little delinting fix that snuck in.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-8-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 7f0a143b0cd7b2b7c05b55b1b6814747ef612ce3
      
https://github.com/qemu/qemu/commit/7f0a143b0cd7b2b7c05b55b1b6814747ef612ce3
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M tests/qemu-iotests/297

  Log Message:
  -----------
  iotests/297: add --namespace-packages to mypy arguments

mypy is kind of weird about how it handles imports. For legacy reasons,
it won't load PEP 420 namespaces, because of logic implemented prior to
that becoming a standard.

So, if you plan on using any, you have to pass
--namespace-packages. Alright, fine.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-9-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: beb6b57b3b1a1fe6ebc208d2edc12b504f69e29f
      
https://github.com/qemu/qemu/commit/beb6b57b3b1a1fe6ebc208d2edc12b504f69e29f
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    A python/.isort.cfg
    R python/qemu/.flake8
    R python/qemu/.isort.cfg
    R python/qemu/__init__.py
    R python/qemu/accel.py
    R python/qemu/console_socket.py
    R python/qemu/machine.py
    A python/qemu/machine/.flake8
    A python/qemu/machine/__init__.py
    A python/qemu/machine/console_socket.py
    A python/qemu/machine/machine.py
    A python/qemu/machine/pylintrc
    A python/qemu/machine/qtest.py
    R python/qemu/pylintrc
    R python/qemu/qmp.py
    A python/qemu/qmp/__init__.py
    R python/qemu/qtest.py
    R python/qemu/utils.py
    A python/qemu/utils/__init__.py
    A python/qemu/utils/accel.py
    M tests/acceptance/avocado_qemu/__init__.py
    M tests/acceptance/virtio-gpu.py
    M tests/qemu-iotests/300
    M tests/qemu-iotests/iotests.py
    M tests/vm/aarch64vm.py
    M tests/vm/basevm.py

  Log Message:
  -----------
  python: create qemu packages

move python/qemu/*.py to python/qemu/[machine, qmp, utils]/*.py and
update import directives across the tree.

This is done to create a PEP420 namespace package, in which we may
create subpackages. To do this, the namespace directory ("qemu") should
not have any modules in it. Those files will go into new 'machine',
'qmp' and 'utils' subpackages instead.

Implement machine/__init__.py making the top-level classes and functions
from its various modules available directly inside the package. Change
qmp.py to qmp/__init__.py similarly, such that all of the useful QMP
library classes are available directly from "qemu.qmp" instead of
"qemu.qmp.qmp".

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-10-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: ea1213b7ccc7c24a3c704dc88bd855df45203fed
      
https://github.com/qemu/qemu/commit/ea1213b7ccc7c24a3c704dc88bd855df45203fed
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    A python/PACKAGE.rst
    A python/setup.cfg
    A python/setup.py

  Log Message:
  -----------
  python: add qemu package installer

Add setup.cfg and setup.py, necessary for installing a package via
pip. Add a ReST document (PACKAGE.rst) explaining the basics of what
this package is for and who to contact for more information. This
document will be used as the landing page for the package on PyPI.

List the subpackages we intend to package by name instead of using
find_namespace because find_namespace will naively also packages tests,
things it finds in the dist/ folder, etc. I could not figure out how to
modify this behavior; adding allow/deny lists to setuptools kept
changing the packaged hierarchy. This works, roll with it.

I am not yet using a pyproject.toml style package manifest, because
"editable" installs are not defined (yet?) by PEP-517/518.

I consider editable installs crucial for development, though they have
(apparently) always been somewhat poorly defined.

Pip now (19.2 and later) now supports editable installs for projects
using pyproject.toml manifests, but might require the use of the
--no-use-pep517 flag, which somewhat defeats the point. Full support for
setup.py-less editable installs was not introduced until pip 21.1.1:
https://github.com/pypa/pip/pull/9547/commits/7a95720e796a5e56481c1cc20b6ce6249c50f357

For now, while the dust settles, stick with the de-facto
setup.py/setup.cfg combination supported by setuptools. It will be worth
re-evaluating this point again in the future when our supported build
platforms all ship a fairly modern pip.

Additional reading on this matter:

https://github.com/pypa/packaging-problems/issues/256
https://github.com/pypa/pip/issues/6334
https://github.com/pypa/pip/issues/6375
https://github.com/pypa/pip/issues/6434
https://github.com/pypa/pip/issues/6438

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-11-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 3afc32906f7bffd8e09b7d247d60b55c49665bd3
      
https://github.com/qemu/qemu/commit/3afc32906f7bffd8e09b7d247d60b55c49665bd3
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    A python/VERSION
    M python/setup.cfg

  Log Message:
  -----------
  python: add VERSION file

Python infrastructure as it exists today is not capable reliably of
single-sourcing a package version from a parent directory. The authors
of pip are working to correct this, but as of today this is not possible.

The problem is that when using pip to build and install a python
package, it copies files over to a temporary directory and performs its
build there. This loses access to any information in the parent
directory, including git itself.

Further, Python versions have a standard (PEP 440) that may or may not
follow QEMU's versioning. In general, it does; but naturally QEMU does
not follow PEP 440. To avoid any automatically-generated conflict, a
manual version file is preferred.

I am proposing:

- Python tooling follows the QEMU version, indirectly, but with a major
  version of 0 to indicate that the API is not expected to be
  stable. This would mean version 0.5.2.0, 0.5.1.1, 0.5.3.0, etc.

- In the event that a Python package needs to be updated independently
  of the QEMU version, a pre-release alpha version should be preferred,
  but *only* after inclusion to the qemu development or stable branches.

  e.g. 0.5.2.0a1, 0.5.2.0a2, and so on should be preferred prior to
  5.2.0's release.

- The Python core tooling makes absolutely no version compatibility
  checks or constraints. It *may* work with releases of QEMU from the
  past or future, but it is not required to.

  i.e., "qemu.machine" will, for now, remain in lock-step with QEMU.

- We reserve the right to split the qemu package into independently
  versioned subpackages at a later date. This might allow for us to
  begin versioning QMP independently from QEMU at a later date, if
  we so choose.

Implement this versioning scheme by adding a VERSION file and setting it
to 0.6.0.0a1.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-12-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 93128815af4efcaba03a5581c959bc7f98ee2725
      
https://github.com/qemu/qemu/commit/93128815af4efcaba03a5581c959bc7f98ee2725
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    A python/README.rst
    A python/qemu/README.rst
    A python/qemu/machine/README.rst
    A python/qemu/qmp/README.rst
    A python/qemu/utils/README.rst

  Log Message:
  -----------
  python: add directory structure README.rst files

Add short readmes to python/, python/qemu/, python/qemu/machine,
python/qemu/qmp, and python/qemu/utils that explain the directory
hierarchy. These readmes are visible when browsing the source on
e.g. gitlab/github and are designed to help new developers/users quickly
make sense of the source tree.

They are not designed for inclusion in a published manual.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-13-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: eae4e442caa087b2ef292a5fb6377236fa8423f2
      
https://github.com/qemu/qemu/commit/eae4e442caa087b2ef292a5fb6377236fa8423f2
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    A python/MANIFEST.in
    M python/README.rst

  Log Message:
  -----------
  python: add MANIFEST.in

When creating a source or binary distribution via 'python3 setup.py
<sdist|bdist>', the VERSION and PACKAGE.rst files aren't bundled by
default. Create a MANIFEST.in file that instructs the build tools to
include these so that installation from these files won't fail.

This is required by 'tox', as well as by the tooling needed to upload
packages to PyPI.

Exclude the 'README.rst' file -- that's intended as a guidebook to our
source tree, not a file that needs to be distributed.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-14-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 41c1d81cf2a9bfdb310576a716f3777e8feb1822
      
https://github.com/qemu/qemu/commit/41c1d81cf2a9bfdb310576a716f3777e8feb1822
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    A python/Pipfile
    M python/README.rst

  Log Message:
  -----------
  python: Add pipenv support

pipenv is a tool used for managing virtual environments with pinned,
explicit dependencies. It is used for precisely recreating python
virtual environments.

pipenv uses two files to do this:

(1) Pipfile, which is similar in purpose and scope to what setup.cfg
lists. It specifies the requisite minimum to get a functional
environment for using this package.

(2) Pipfile.lock, which is similar in purpose to `pip freeze >
requirements.txt`. It specifies a canonical virtual environment used for
deployment or testing. This ensures that all users have repeatable
results.

The primary benefit of using this tool is to ensure *rock solid*
repeatable CI results with a known set of packages. Although I endeavor
to support as many versions as I can, the fluid nature of the Python
toolchain often means tailoring code for fairly specific versions.

Note that pipenv is *not* required to install or use this module; this is
purely for the sake of repeatable testing by CI or developers.

Here, a "blank" pipfile is added with no dependencies, but specifies
Python 3.6 for the virtual environment.

Pipfile will specify our version minimums, while Pipfile.lock specifies
an exact loadout of packages that were known to operate correctly. This
latter file provides the real value for easy setup of container images
and CI environments.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-15-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: d1e0476958cd275419754b8acf31a9f1dc62d3dd
      
https://github.com/qemu/qemu/commit/d1e0476958cd275419754b8acf31a9f1dc62d3dd
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M python/qemu/machine/__init__.py
    M python/qemu/machine/machine.py
    M python/qemu/machine/qtest.py

  Log Message:
  -----------
  python: add pylint import exceptions

Pylint 2.5.x - 2.7.x have regressions that make import checking
inconsistent, see:

https://github.com/PyCQA/pylint/issues/3609
https://github.com/PyCQA/pylint/issues/3624
https://github.com/PyCQA/pylint/issues/3651

Pinning to 2.4.4 is worse, because it mandates versions of shared
dependencies that are too old for features we want in isort and mypy.
Oh well.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-16-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: ef42440d797a1549dd64fe2a51500ba55fe54c3f
      
https://github.com/qemu/qemu/commit/ef42440d797a1549dd64fe2a51500ba55fe54c3f
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    R python/qemu/machine/pylintrc
    M python/setup.cfg

  Log Message:
  -----------
  python: move pylintrc into setup.cfg

Delete the empty settings now that it's sharing a home with settings for
other tools.

pylint can now be run from this folder as "pylint qemu".

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-17-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: b4d37d8188dbff34f0bf88279eeb5b6cb6d1ff82
      
https://github.com/qemu/qemu/commit/b4d37d8188dbff34f0bf88279eeb5b6cb6d1ff82
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M python/Pipfile
    A python/Pipfile.lock

  Log Message:
  -----------
  python: add pylint to pipenv

We are specifying >= pylint 2.8.x for several reasons:

1. For setup.cfg support, added in pylint 2.5.x
2. To specify a version that has incompatibly dropped
   bad-whitespace checks (2.6.x)
3. 2.7.x fixes "unsubscriptable" warnings in Python 3.9
4. 2.8.x adds a new, incompatible 'consider-using-with'
   warning that must be disabled in some cases.
   These pragmas cause warnings themselves in 2.7.x.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-18-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 81f8c4467c1899ef1ba984c70c328ac0c32af10c
      
https://github.com/qemu/qemu/commit/81f8c4467c1899ef1ba984c70c328ac0c32af10c
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    R python/qemu/machine/.flake8
    M python/setup.cfg

  Log Message:
  -----------
  python: move flake8 config to setup.cfg

Update the comment concerning the flake8 exception to match commit
42c0dd12, whose commit message stated:

A note on the flake8 exception: flake8 will warn on *any* bare except,
but pylint's is context-aware and will suppress the warning if you
re-raise the exception.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-19-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 21d0b8667981e386cdfec18ad7d3eb4d9a33b088
      
https://github.com/qemu/qemu/commit/21d0b8667981e386cdfec18ad7d3eb4d9a33b088
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M python/setup.cfg

  Log Message:
  -----------
  python: add excluded dirs to flake8 config

Instruct flake8 to avoid certain well-known directories created by
python tooling that it ought not check.

Note that at-present, nothing actually creates a ".venv" directory; but
it is in such widespread usage as a de-facto location for a developer's
virtual environment that it should be excluded anyway. A forthcoming
commit canonizes this with a "make venv" command.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-20-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 6d17d910434568626c1c739b7d3d8433618b19eb
      
https://github.com/qemu/qemu/commit/6d17d910434568626c1c739b7d3d8433618b19eb
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M python/Pipfile
    M python/Pipfile.lock

  Log Message:
  -----------
  python: Add flake8 to pipenv

flake8 3.5.x does not support the --extend-ignore syntax used in the
.flake8 file to gracefully extend default ignores, so 3.6.x is our
minimum requirement. There is no known upper bound.

flake8 can be run from the python/ directory with no arguments.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-21-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: e941c844e4446b6200ac102ef285544c406a2fcd
      
https://github.com/qemu/qemu/commit/e941c844e4446b6200ac102ef285544c406a2fcd
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    R python/mypy.ini
    M python/setup.cfg

  Log Message:
  -----------
  python: move mypy.ini into setup.cfg

mypy supports reading its configuration values from a central project
configuration file; do so.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-22-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 0542a4c95767b2370cb6622efe723bb6197aa04c
      
https://github.com/qemu/qemu/commit/0542a4c95767b2370cb6622efe723bb6197aa04c
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M python/Pipfile
    M python/Pipfile.lock
    M python/setup.cfg

  Log Message:
  -----------
  python: add mypy to pipenv

0.730 appears to be about the oldest version that works with the
features we want, including nice human readable output (to make sure
iotest 297 passes), and type-parameterized Popen generics.

0.770, however, supports adding 'strict' to the config file, so require
at least 0.770.

Now that we are checking a namespace package, we need to tell mypy to
allow PEP420 namespaces, so modify the mypy config as part of the move.

mypy can now be run from the python root by typing 'mypy -p qemu'.

A note on mypy invocation: Running it as "mypy qemu/" changes the import
path detection mechanisms in mypy slightly, and it will fail. See
https://github.com/python/mypy/issues/8584 for a decent entry point with
more breadcrumbs on the various behaviors that contribute to this subtle
difference.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-23-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 158ac451b9e1029798f8fdc103fef64830e4314e
      
https://github.com/qemu/qemu/commit/158ac451b9e1029798f8fdc103fef64830e4314e
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    R python/.isort.cfg
    M python/setup.cfg

  Log Message:
  -----------
  python: move .isort.cfg into setup.cfg

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-24-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 22a973cb1d365f6c506e190d26e2261a65066e15
      
https://github.com/qemu/qemu/commit/22a973cb1d365f6c506e190d26e2261a65066e15
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M python/Pipfile
    M python/Pipfile.lock

  Log Message:
  -----------
  python/qemu: add isort to pipenv

isort 5.0.0 through 5.0.4 has a bug that causes it to misinterpret
certain "from ..." clauses that are not related to imports.

isort < 5.1.1 has a bug where it does not handle comments near import
statements correctly.

Require 5.1.2 or greater.

isort can be run (in "check" mode) with 'isort -c qemu' from the python
root. isort can also be used to fix/rewrite import order automatically
by using 'isort qemu'.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-25-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: a4dd49d40536b7ad70ab9c2e25a7810773ca32bc
      
https://github.com/qemu/qemu/commit/a4dd49d40536b7ad70ab9c2e25a7810773ca32bc
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M python/Pipfile
    M python/Pipfile.lock

  Log Message:
  -----------
  python/qemu: add qemu package itself to pipenv

This adds the python qemu packages themselves to the pipenv manifest.
'pipenv sync' will create a virtual environment sufficient to use the SDK.
'pipenv sync --dev' will create a virtual environment sufficient to use
and test the SDK (with pylint, mypy, isort, flake8, etc.)

The qemu packages are installed in 'editable' mode; all changes made to
the python package inside the git tree will be reflected in the
installed package without reinstallation. This includes changes made
via git pull and so on.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-26-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: dbe75f55669a4e2295b0dae161b8f796e6dbaded
      
https://github.com/qemu/qemu/commit/dbe75f55669a4e2295b0dae161b8f796e6dbaded
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M python/PACKAGE.rst
    M python/Pipfile
    M python/Pipfile.lock
    M python/README.rst
    M python/setup.cfg

  Log Message:
  -----------
  python: add devel package requirements to setuptools

setuptools doesn't have a formal understanding of development requires,
but it has an optional feataures section. Fine; add a "devel" feature
and add the requirements to it.

To avoid duplication, we can modify pipenv to install qemu[devel]
instead. This enables us to run invocations like "pip install -e
.[devel]" and test the package on bleeding-edge packages beyond those
specified in Pipfile.lock.

Importantly, this also allows us to install the qemu development
packages in a non-networked mode: `pip3 install --no-index -e .[devel]`
will now fail if the proper development dependencies are not already
met. This can be useful for automated build scripts where fetching
network packages may be undesirable.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-27-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 31622b2a8ac769b3cef730d3a24ed209e3861cbc
      
https://github.com/qemu/qemu/commit/31622b2a8ac769b3cef730d3a24ed209e3861cbc
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M python/Pipfile.lock
    M python/README.rst
    A python/avocado.cfg
    M python/setup.cfg
    A python/tests/flake8.sh
    A python/tests/isort.sh
    A python/tests/mypy.sh
    A python/tests/pylint.sh

  Log Message:
  -----------
  python: add avocado-framework and tests

Try using avocado to manage our various tests; even though right now
they're only invoking shell scripts and not really running any
python-native code.

Create tests/, and add shell scripts which call out to mypy, flake8,
pylint and isort to enforce the standards in this directory.

Add avocado-framework to the setup.cfg development dependencies, and add
avocado.cfg to store some preferences for how we'd like the test output
to look.

Finally, add avocado-framework to the Pipfile environment and lock the
new dependencies. We are using avocado >= 87.0 here to take advantage of
some features that Cleber has helpfully added to make the test output
here *very* friendly and easy to read for developers that might chance
upon the output in Gitlab CI.

[Note: ALL of the dependencies get updated to the most modern versions
that exist at the time of this writing. No way around it that I have
seen. Not ideal, but so it goes.]

Provided you have the right development dependencies (mypy, flake8,
isort, pylint, and now avocado-framework) You should be able to run
"avocado --config avocado.cfg run tests/" from the python folder to run
all of these linters with the correct arguments.

(A forthcoming commit adds the much easier 'make check'.)

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-28-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 6560379facf40e66fd8fbf4578f3d28f510167d8
      
https://github.com/qemu/qemu/commit/6560379facf40e66fd8fbf4578f3d28f510167d8
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    A python/Makefile
    M python/PACKAGE.rst
    M python/README.rst

  Log Message:
  -----------
  python: add Makefile for some common tasks

Add "make venv" to create the pipenv-managed virtual environment that
contains our explicitly pinned dependencies.

Add "make check" to run the python linters [in the host execution
environment].

Add "make venv-check" which combines the above two: create/update the
venv, then run the linters in that explicitly managed environment.

Add "make develop" which canonizes the runes needed to get both the
linting pre-requisites (the "[devel]" part), and the editable
live-install (the "-e" part) of these python libraries.

make clean: delete miscellaneous python packaging output possibly
created by pipenv, pip, or other python packaging utilities

make distclean: delete the above, the .venv, and the editable "qemu"
package forwarder (qemu.egg-info) if there is one.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-29-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: f9c0600f0200528921c43ccb8a8a44c81825a343
      
https://github.com/qemu/qemu/commit/f9c0600f0200528921c43ccb8a8a44c81825a343
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    A python/.gitignore

  Log Message:
  -----------
  python: add .gitignore

Ignore *Python* build and package output (build, dist, qemu.egg-info);
these files are not created as part of a QEMU build. They are created by
running the commands 'python3 setup.py <sdist|bdist>' when preparing
tarballs to upload to e.g. PyPI.

Ignore miscellaneous cached python confetti (mypy, pylint, et al)

Ignore .idea (pycharm) .vscode, and .venv (pipenv et al).

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-30-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 3c8de38c8515a300b7842d95893b9e95caaa0ad6
      
https://github.com/qemu/qemu/commit/3c8de38c8515a300b7842d95893b9e95caaa0ad6
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M python/.gitignore
    M python/Makefile
    M python/setup.cfg

  Log Message:
  -----------
  python: add tox support

This is intended to be a manually run, non-CI script.

Use tox to test the linters against all python versions from 3.6 to
3.10. This will only work if you actually have those versions installed
locally, but Fedora makes this easy:

> sudo dnf install python3.6 python3.7 python3.8 python3.9 python3.10

Unlike the pipenv tests (make venv-check), this pulls "whichever"
versions of the python packages, so they are unpinned and may break as
time goes on. In the case that breakages are found, setup.cfg should be
amended accordingly to avoid the bad dependant versions, or the code
should be amended to work around the issue.

With confidence that the tests pass on 3.6 through 3.10 inclusive, add
the appropriate classifiers to setup.cfg to indicate which versions we
claim to support.

Tox 3.18.0 or above is required to use the 'allowlist_externals' option.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-31-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 6b9c277797879ce41ed20deb6737f4156cc279b3
      
https://github.com/qemu/qemu/commit/6b9c277797879ce41ed20deb6737f4156cc279b3
  Author: John Snow <jsnow@redhat.com>
  Date:   2021-06-01 (Tue, 01 Jun 2021)

  Changed paths:
    M .gitlab-ci.d/containers.yml
    M .gitlab-ci.d/static_checks.yml
    A tests/docker/dockerfiles/python.docker

  Log Message:
  -----------
  gitlab: add python linters to CI

Add a Python container that has just enough juice for us to run the
Python code quality analysis tools. Base this container on Fedora,
because Fedora has very convenient packaging for testing multiple Python
versions.

We need python3, pip (for pulling packages), pipenv and virtualenv for
creating virtual environments, and tox for running tests. make is needed
for running 'make check-tox' and 'make venv-check' targets. Python3.10
is needed explicitly because the tox package only pulls in 3.6-3.9, but
we wish to test the forthcoming release of Python as well to help
predict any problems. Lastly, we need gcc to compile PyPI packages that
may not have a binary distribution available.

Add two tests:

check-python-pipenv uses pipenv to test a frozen, very explicit set of
packages against our minimum supported python version, Python 3.6. This
test is not allowed to fail. The dependencies this test uses do not
change unless python/Pipfile.lock is changed.

check-python-tox uses tox to install the latest versions of required
python dependencies against a wide array of Python versions from 3.6 to
3.9, even including the yet-to-be-released Python 3.10. This test is
allowed to fail with a warning.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20210527211715.394144-32-jsnow@redhat.com
[Fix rebase conflict over .gitlab-ci.yml --js]
Signed-off-by: John Snow <jsnow@redhat.com>


  Commit: 49ba51adec7928fe7cf3cb43acbf0b953e5c637e
      
https://github.com/qemu/qemu/commit/49ba51adec7928fe7cf3cb43acbf0b953e5c637e
  Author: Peter Maydell <peter.maydell@linaro.org>
  Date:   2021-06-02 (Wed, 02 Jun 2021)

  Changed paths:
    M .gitlab-ci.d/containers.yml
    M .gitlab-ci.d/static_checks.yml
    M docs/devel/testing.rst
    A python/.gitignore
    A python/MANIFEST.in
    A python/Makefile
    A python/PACKAGE.rst
    A python/Pipfile
    A python/Pipfile.lock
    A python/README.rst
    A python/VERSION
    A python/avocado.cfg
    R python/mypy.ini
    R python/qemu/.flake8
    R python/qemu/.isort.cfg
    A python/qemu/README.rst
    R python/qemu/__init__.py
    R python/qemu/accel.py
    R python/qemu/console_socket.py
    R python/qemu/machine.py
    A python/qemu/machine/README.rst
    A python/qemu/machine/__init__.py
    A python/qemu/machine/console_socket.py
    A python/qemu/machine/machine.py
    A python/qemu/machine/qtest.py
    R python/qemu/pylintrc
    R python/qemu/qmp.py
    A python/qemu/qmp/README.rst
    A python/qemu/qmp/__init__.py
    R python/qemu/qtest.py
    A python/qemu/utils/README.rst
    A python/qemu/utils/__init__.py
    A python/qemu/utils/accel.py
    A python/setup.cfg
    A python/setup.py
    A python/tests/flake8.sh
    A python/tests/isort.sh
    A python/tests/mypy.sh
    A python/tests/pylint.sh
    M tests/acceptance/avocado_qemu/__init__.py
    M tests/acceptance/boot_linux.py
    A tests/acceptance/hotplug_cpu.py
    A tests/acceptance/info_usernet.py
    M tests/acceptance/linux_ssh_mips_malta.py
    M tests/acceptance/virtio-gpu.py
    M tests/acceptance/virtiofs_submounts.py
    A tests/docker/dockerfiles/python.docker
    M tests/qemu-iotests/297
    M tests/qemu-iotests/300
    M tests/qemu-iotests/iotests.py
    M tests/requirements.txt
    M tests/vm/aarch64vm.py
    M tests/vm/basevm.py

  Log Message:
  -----------
  Merge remote-tracking branch 'remotes/jsnow-gitlab/tags/python-pull-request' 
into staging

Pull request

V2:
 - Squashed in fixup for
   'Python: add utility function for retrieving port redirection'
 - Rebased on today's upstream

CI here:
https://gitlab.com/jsnow/qemu/-/pipelines/313202814

# gpg: Signature made Wed 02 Jun 2021 00:29:55 BST
# gpg:                using RSA key F9B7ABDBBCACDF95BE76CBD07DEF8106AAFC390E
# gpg: Good signature from "John Snow (John Huston) <jsnow@redhat.com>" [full]
# Primary key fingerprint: FAEB 9711 A12C F475 812F  18F2 88A9 064D 1835 61EB
#      Subkey fingerprint: F9B7 ABDB BCAC DF95 BE76  CBD0 7DEF 8106 AAFC 390E

* remotes/jsnow-gitlab/tags/python-pull-request: (44 commits)
  gitlab: add python linters to CI
  python: add tox support
  python: add .gitignore
  python: add Makefile for some common tasks
  python: add avocado-framework and tests
  python: add devel package requirements to setuptools
  python/qemu: add qemu package itself to pipenv
  python/qemu: add isort to pipenv
  python: move .isort.cfg into setup.cfg
  python: add mypy to pipenv
  python: move mypy.ini into setup.cfg
  python: Add flake8 to pipenv
  python: add excluded dirs to flake8 config
  python: move flake8 config to setup.cfg
  python: add pylint to pipenv
  python: move pylintrc into setup.cfg
  python: add pylint import exceptions
  python: Add pipenv support
  python: add MANIFEST.in
  python: add directory structure README.rst files
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>


Compare: https://github.com/qemu/qemu/compare/dd2db39d7843...49ba51adec79



reply via email to

[Prev in Thread] Current Thread [Next in Thread]