Re: [PATCH 1/2] Acceptance Tests: restore downloading of VM images

[Philippe Mathieu-Daudé]

On 2/27/21 12:04 AM, Cleber Rosa wrote:
> On Fri, Feb 26, 2021 at 01:01:28AM +0100, Philippe Mathieu-Daudé wrote:
>> On 2/26/21 12:21 AM, Cleber Rosa wrote:
>>> The "get-vm-images" target defined in tests/Makefile.include is a
>>> prerequisite for "check-acceptance", so that those files get
>>> downloaded before the Avocado job even starts.
>>>
>>> It looks like on c401c058a1c a TARGETS variable was introduced with a
>>> different content than it was previously coming from the main
>>> Makefile.  From that point on, the "get-vm-images" succeed without
>>> doing anything because there was no matching architecture to download.
>>
>> Any idea about how to detect such side effects (tests silently
>> disabled) automatically?
>>
> 
> It wasn't really that any tests were disabled... they all continued to
> run.  In this case it was a broken make rule that caused the download
> of the images, ahead of time, to not be performed.
> 
> But your question is still valid and something that could happen.  The
> best answer I have is that all job results could and should also be
> persisted in a structured way that is succeptible to being queried.
> Then on top of that, you can build queries to show stability metrics,
> regressions, etc.
> 
> To that regards, I can speak about three possibilities:
> 
> 1) Avocado has support for Fedora's resultsdb[1][2]
> 
> 2) Because the Acceptance tests are already communicating the test
> results to GitLab (via junit), using the GitLab API that lets you
> query the detailed test results
> 
> 3) In addition to that, Marcelo (cc'd here) has written an Avocado plugin
> that will export test resutls suitable to be used on a datawarehouse
> tool developed by the Continuous Kernel Integration project[3]. This
> is not generally available at the moment, but should be available
> soon.
> 
> Regards,
> - Cleber.
> 
> [1] - https://taskotron.fedoraproject.org/resultsdb/results
> [2] - 
> https://avocado-framework.readthedocs.io/en/85.0/plugins/optional/results.html#resultsdb-plugin
> [3] - https://cki-project.org

Wow this is thrilling!

Maybe we could use fosshost to run a resultsdb VM.