Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Cleber Rosa]

On Thu, Aug 29, 2019 at 11:51:17AM +1000, David Gibson wrote:
> On Thu, Aug 29, 2019 at 11:31:25AM +1000, David Gibson wrote:
> > If I attempt to run "make check-acceptance" on my POWER9, RHEL8.1
> > machine when the openssl-devel package isn't installed, I get the
> > following very cryptic error:
> > 
> >   VENV    /home/dwg/qemu/build/rhel8/tests/venv
> >   PIP     /home/dwg/qemu/tests/requirements.txt
> > Command "/home/dwg/qemu/build/rhel8/tests/venv/bin/python -u -c "import 
> > setuptools, 
> > tokenize;__file__='/tmp/pip-build-la4el5r5/cryptography/setup.py';f=getattr(tokenize,
> >  'open', open)(__file__);code=f.read().replace('\r\n', 
> > '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record 
> > /tmp/pip-1efs22iz-record/install-record.txt 
> > --single-version-externally-managed --compile --install-headers 
> > /home/dwg/qemu/build/rhel8/tests/venv/include/site/python3.6/cryptography" 
> > failed with error code 1 in /tmp/pip-build-la4el5r5/cryptography/
> > 
> > Using V=1 doesn't give any more useful information, and it's not
> > (easily) possible to manually re-run the given command since it relies
> > on things in /tmp that are removed once the attempt finishes.
> > 
> > I only figured out it was openssl-devel being missing that was the
> > problem by (mostly) guesswork.  It would be really great if we could
> > generate a more helpful error here.
> > 
> > In addition, if I rerun "make check-acceptance" it no longer even
> > attempts the PIP install, since tests/venv already exists in my build
> > environment.  It then sort of works, but I think it might be hitting
> > other errors because of the missing python packages.  Sorry that's a
> > bit vague - I also seem to be getting unrelated errors that I'm still
> > trying to figure out.
> 
> Fwiw, I also get an equally cryptic error that I haven't figured out
> the cause for on my 32-bit Fedora container environment:
> 
>   VENV    /home/dwg/src/qemu/build/i386/tests/venv
>   PIP     /home/dwg/src/qemu/tests/requirements.txt
>   Failed building wheel for bcrypt
> Could not build wheels for bcrypt which use PEP 517 and cannot be installed 
> directly
> You are using pip version 19.0.3, however version 19.2.3 is available.
> You should consider upgrading via the 'pip install --upgrade pip' command.
>

This is certainly caused by pip not being able to install paramiko on
those systems.  I have dealt with paramiko (and its many dependencies)
before on the avocado remote runner plugin (which is not being used
here) and it was not fun.

My personal goal was to rely on the ssh binary as an ssh client, which
should be more ubiquitous, and with that I added a simple wrapper to
Avocado:

  
https://avocado-framework.readthedocs.io/en/71.0/api/utils/avocado.utils.html#module-avocado.utils.ssh

I guess we should consider changing the (few) tests that require
paramiko to use that module instead.

> In this case the check definitely doesn't work - it doesn't appear to
> have installed avocado in the venv.
> 
>   AVOCADO tests/acceptance
> /home/dwg/src/qemu/build/i386/tests/venv/bin/python: No module named avocado
>

My hope is that with pure Python modules in requirements.txt, failures
will be much harder to come by.  Either way, it'd be nice to improve
the venv creation, at the very least add a cleanup if it fails to
complete successfully.

As a workaround I'd suggest two things:

 1) remove paramiko from requirements.txt
 2) set the env var CONTINUOUS_INTEGRATION=1 before running `make
    check-acceptace` (the tests in linux_ssh_mips_malta.py check for
    that and don't run if it's set).

But, we'll need to address those failures definitely ASAP.  Anyway,
thanks for reporting it.

- Cleber.

> -- 
> David Gibson                  | I'll have my music baroque, and my code
> david AT gibson.dropbear.id.au        | minimalist, thank you.  NOT _the_ 
> _other_
>                               | _way_ _around_!
> http://www.ozlabs.org/~dgibson