Re: [Qemu-devel] Cryptic errors from PIP install if missing openssl-devel

[Philippe Mathieu-Daudé]

On 8/29/19 5:27 AM, Cleber Rosa wrote:
> On Thu, Aug 29, 2019 at 11:51:17AM +1000, David Gibson wrote:
>> On Thu, Aug 29, 2019 at 11:31:25AM +1000, David Gibson wrote:
>>> If I attempt to run "make check-acceptance" on my POWER9, RHEL8.1
>>> machine when the openssl-devel package isn't installed, I get the
>>> following very cryptic error:
>>>
>>>   VENV    /home/dwg/qemu/build/rhel8/tests/venv
>>>   PIP     /home/dwg/qemu/tests/requirements.txt
>>> Command "/home/dwg/qemu/build/rhel8/tests/venv/bin/python -u -c "import 
>>> setuptools, 
>>> tokenize;__file__='/tmp/pip-build-la4el5r5/cryptography/setup.py';f=getattr(tokenize,
>>>  'open', open)(__file__);code=f.read().replace('\r\n', 
>>> '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record 
>>> /tmp/pip-1efs22iz-record/install-record.txt 
>>> --single-version-externally-managed --compile --install-headers 
>>> /home/dwg/qemu/build/rhel8/tests/venv/include/site/python3.6/cryptography" 
>>> failed with error code 1 in /tmp/pip-build-la4el5r5/cryptography/
>>>
>>> Using V=1 doesn't give any more useful information, and it's not
>>> (easily) possible to manually re-run the given command since it relies
>>> on things in /tmp that are removed once the attempt finishes.
>>>
>>> I only figured out it was openssl-devel being missing that was the
>>> problem by (mostly) guesswork.  It would be really great if we could
>>> generate a more helpful error here.
>>>
>>> In addition, if I rerun "make check-acceptance" it no longer even
>>> attempts the PIP install, since tests/venv already exists in my build
>>> environment.  It then sort of works, but I think it might be hitting
>>> other errors because of the missing python packages.  Sorry that's a
>>> bit vague - I also seem to be getting unrelated errors that I'm still
>>> trying to figure out.
>>
>> Fwiw, I also get an equally cryptic error that I haven't figured out
>> the cause for on my 32-bit Fedora container environment:
>>
>>   VENV    /home/dwg/src/qemu/build/i386/tests/venv
>>   PIP     /home/dwg/src/qemu/tests/requirements.txt
>>   Failed building wheel for bcrypt
>> Could not build wheels for bcrypt which use PEP 517 and cannot be installed 
>> directly
>> You are using pip version 19.0.3, however version 19.2.3 is available.
>> You should consider upgrading via the 'pip install --upgrade pip' command.
>>
> 
> This is certainly caused by pip not being able to install paramiko on
> those systems.  I have dealt with paramiko (and its many dependencies)
> before on the avocado remote runner plugin (which is not being used
> here) and it was not fun.
> 
> My personal goal was to rely on the ssh binary as an ssh client, which
> should be more ubiquitous, and with that I added a simple wrapper to
> Avocado:
> 
>   
> https://avocado-framework.readthedocs.io/en/71.0/api/utils/avocado.utils.html#module-avocado.utils.ssh

I did not know this module.

> I guess we should consider changing the (few) tests that require
> paramiko to use that module instead.

Clean way to resolve this issue.

>> In this case the check definitely doesn't work - it doesn't appear to
>> have installed avocado in the venv.
>>
>>   AVOCADO tests/acceptance
>> /home/dwg/src/qemu/build/i386/tests/venv/bin/python: No module named avocado
>>
> 
> My hope is that with pure Python modules in requirements.txt, failures
> will be much harder to come by.  Either way, it'd be nice to improve

Is there a way to check for this before submitting/merging patches?

Some checkpatch.py lines that report:

  The 'paramiko' module is not pure Python and can
  not be added in requirements.txt.

> the venv creation, at the very least add a cleanup if it fails to
> complete successfully.
> 
> As a workaround I'd suggest two things:
> 
>  1) remove paramiko from requirements.txt
>  2) set the env var CONTINUOUS_INTEGRATION=1 before running `make
>     check-acceptace` (the tests in linux_ssh_mips_malta.py check for
>     that and don't run if it's set).
> 
> But, we'll need to address those failures definitely ASAP.  Anyway,
> thanks for reporting it.
> 
> - Cleber.
> 
>> -- 
>> David Gibson                 | I'll have my music baroque, and my code
>> david AT gibson.dropbear.id.au       | minimalist, thank you.  NOT _the_ 
>> _other_
>>                              | _way_ _around_!
>> http://www.ozlabs.org/~dgibson
> 
> 
>