Re: Should we keep using Avocado for functional testing? (was: Re: [PULL 22/23] hw/sd: Fix incorrect populated function switch status data structure)

[Cleber Rosa]

On Fri, Oct 23, 2020 at 11:34:29AM +0200, Philippe Mathieu-Daudé wrote:
> On 10/23/20 11:23 AM, Philippe Mathieu-Daudé wrote:
> 
> Back to the cache problem, I started to ask 2 years ago,
> https://www.mail-archive.com/avocado-devel@redhat.com/msg00860.html
> note this is the exact situation we are having:
>

Hi Phil,

I think the issue on this thread was more on the server mirror side,
than on the *cache* side of things, right?  Still, it's an open issue
that needs to be discussed indeed.

> >> - What will happens if I add tests downloading running on their
> >> compiled u-boot
> >> (https://downloads.gumstix.com/images/angstrom/developer/2012-01-22-1750/u-boot.bin)
> >> and the company decides to remove this old directory?
> >> Since sometimes old open-source software are hard to rebuild with
> >> recent compilers, should we consider to use a public storage to keep
> >> open-source (signed) blobs we can use for integration testing?
> >
> > For Avocado-VT, there are the JeOS images[1], which we keep on a test
> > "assets" directory.  We have a lot of storage/bandwidth availability,
> > so it can be used for other assets proven to be necessary for tests.
> >
> > As long as distribution rights and licensing are not issues, we can
> > definitely use the same server for kernels, u-boot images and what
> > not.
> >
> > [1] - https://avocado-project.org/data/assets/
> 
> 
> We discussed about this again last year at the KVM forum. Various
> RFE have been filled:
> https://www.mail-archive.com/avocado-devel@redhat.com/msg01183.html
>

I understand it has been a year, which is a lot of time specially when
it comes to an urgent need such as more and better testing and CI.
But some things can be patched quickly, while others need further
structural work.  More on that later...

> 
> What we need is a QEMU community file server similar to the asset
> one used by the Avocado community. The problem is who is going to
> pay and sysadmin it. IIRC Gerd suggested to use GitLab Page, but
> the artifact file size is limited to a maximum of 1GiB:
> https://docs.gitlab.com/ee/user/gitlab_com/#gitlab-pages
> Alternative is to use a git-lfs server:
> https://docs.gitlab.com/ee/topics/git/lfs/#hosting-lfs-objects-externally
> This is now a project management problem.
> 
> Next week is the KVM forum 2020 and there might be a BoF session
> to talk about that:
> https://kvmforum2020.sched.com/overview/type/Birds+of+a+Feather+Sessions+(BoF)
> 
> 
> Now that the QEMU community started to use gitlab-ci more and more
> I received complains that Avocado is not practical (hard to
> understand how to reproduce command line to debug, options to use
> not clear, timeouts not clear, why download all artifacts to run
> a single test, various issues regarding caching, cache filled /home
> filesystem) so I have been asked to look at Avocado alternatives,
> because we want contributors run more tests and CI, not them be
> scared by it.

What I can say is that the Avocado developers have been working
on a major overhaul to address those issues.  I'm not sure this
is the place/time to go over each of those items, but there may be
some confusion going on some of those issues.  For instance, your
point about "why download all artifacts to run a single test".
It shouldn't, and it doesn't:

  $ du -hcs ~/avocado/data/cache/
  4.0K    /home/cleber/avocado/data/cache/
  4.0K    total

  $ ./tests/venv/bin/avocado run 
tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_x86_64_pc 
  Fetching asset from 
tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_x86_64_pc
  JOB ID     : a3c08d7a0acc613c207b27e4b291a0010965af2d
  JOB LOG    : 
/home/cleber/avocado/job-results/job-2020-10-26T11.50-a3c08d7/job.log
   (1/1) 
tests/acceptance/boot_linux_console.py:BootLinuxConsole.test_x86_64_pc: PASS 
(2.04 s)
  RESULTS    : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
CANCEL 0
  JOB TIME   : 3.22 s

  $ du -hcs ~/avocado/data/cache/
  8.3M    /home/cleber/avocado/data/cache/
  8.3M    total

About Avocado being hard to understand, I'm a bit saddened by that,
because the team spent a lot of effort in:

 * Restructuring / rewriting / reviewing the documention
 * Normalizing the command line options
 * Deprecating and removing a lot of non-relevant functionality

I've probably said the same thing in a few different channels and
contexts, but for the last ~9 months, the Avocado project had been
working on a new architecture that was highly influenced by the
limitations we found while using Avocado in QEMU.  Version 82.0 was
the result of that work, and it'd be very bad timing if, instead of
integrating this "new Avocado", QEMU would start "from scratch" again.

I've been looking for an opportunity to present what this "new
Avocado" is.  Maybe this or another BoF would be a good place?

> I haven't looked at alternatives yet, from the various features
> we could have use, the biggest one is the ability to interact with
> the serial console. And that feature is duplicated with the VM
> testing class: tests/vm/basevm.py.
> The SSH session used in linux_ssh_mips_malta.py is similar to the
> one from basevm.py too (see 'make vm-boot-ssh-%').
> Some tests are not 'acceptance' but simple qtest written in Python,
> such cpu_queries.py / pc_cpu_hotplug_props.py / migration.py /
> pc_cpu_hotplug_props.py / x86_cpu_model_versions.py.
> 
> The classes I find practical are downloading artifact, uncompressing
> or extracting archive, and eventually the cloudinit one. But we can probably
> use them directly.
>

I understand this is your PoV with regards to how much value you
believe exists in those particular utility modules.  The challenge is
how to combine other people's PoV with regards to the same matter, and
manage them accordingly as a group.  One example is how Pavel Dovgaluk
may find the "gdb" utility module important.

Now consider that Red Hat is vested to have some of the functional /
integration tests that run downstream (and which are based on
Avocado+Avocado-VT) migrated upstreamed.  Then, all of a sudden, a
much larger group of "practical classes" arise.

Finally, these are, as you mentioned in another thread, utility
libraries (such as python/qemu) are commonly needed in other projects
(specially those related to QEMU).  I understand that suggesting to
have more code infrastructure code living in the QEMU would actually
make it harder to solve similar problems once.

> 
> Lot of material to discuss :)
>

Absolutely!

Best,
- Cleber.

> Regards,
> 
> Phil.
>

signature.asc
Description: PGP signature