Re: [PATCH v3 06/10] iotests: add testfinder.py

[Vladimir Sementsov-Ogievskiy]

08.05.2020 11:49, Kevin Wolf wrote:
> Am 07.05.2020 um 19:43 hat Vladimir Sementsov-Ogievskiy geschrieben:
> > 21.04.2020 19:56, Kevin Wolf wrote:
> > > Am 21.04.2020 um 09:35 hat Vladimir Sementsov-Ogievskiy geschrieben:
> > > > Add python script with new logic of searching for tests:
> > > >
> > > > Current ./check behavior:
> > > >    - tests are named [0-9][0-9][0-9]
> > > >    - tests must be registered in group file (even if test doesn't belong
> > > >      to any group, like 142)
> > > >
> > > > Behavior of new test:
> > > >    - group file is dropped
> > > >    - tests are searched by file-name instead of group file, so it's not
> > > >      needed more to "register the test", just create it with name
> > > >      *-test. Old names like [0-9][0-9][0-9] are supported too, but not
> > > >      recommended for new tests
> > >
> > > I wonder if a tests/ subdirectory instead of the -test suffix would
> > > organise things a bit better.
> >
> > It will make more difficult to import iotests.py.. Calling common.rc from
> > bash tests will need to be modified too.
> >
> > So, we'll need additional line in all python tests:
> >
> > sys.path.append(os.path.join(os.path.dirname(__file__), '..'))
>
> Hm, yes, this is nasty.
>
> Would it be any better have a wrapper script that just imports the
> actual test so that the import path would still contain the main
> qemu-iotests/ directory?

I think, that better to keep all tests self-executable without any additional 
preparations, just set all interface environment variables and run test (or use 
check to automate environment initialization). In this way we can support any 
executable, not even limited to  python and bash (I doubt that we need more, 
but keeping test interface full-defined by environment variables seems a good 
thing).

> > which doesn't seem to be good practice.. So, instead we'd better call
> > tests with PYTHONPATH set appropriately..
>
> This is another option, especially if we do want to create a lib/.
>
> > and modify bash tests to do
> > . ../common.rc
> > . ../common.filter
> > <etc>
> >
> > or again, better to export BASH_TEST_LIB directory.
>
> I don't think this is necessary because the working directory wouldn't
> change, so bash scripts should just keep working as before.

Hmm, right. Probably same works for python tests as well?

> > Is it worth doing?
> >
> > I think, there are two variants:
> >
> > 1) keep as is: all in one directory, add *-test notation
>
> I think it would make it rather hard to find the files that belong to
> the test harness implementation between all the tests. Currently, 'ls'
> in the qemu-iotests directory is still kind of usable because all the
> test cases are at the start and everything that comes later is not a
> test.
>
> > 2) go further and restructure to something like:
> >
> > iotests/<only check script and its dependencies here>
> > iotests/tests/<all-tests-here>
> > iotests/lib/python/iotests.py
> > iotests/lib/bash/<common.rc and friends here>
> >
> > And then, check script will export PYTHONPATH and BASH_TEST_LIB
> > variables.
>
> I think leaving everything except named tests where it is, but setting
> PYTHONPATH or having a wrapper script, is still a third option that's
> worth considering. It sounds like the most attrative option to me.

Works for me too. OK, let's go this way.

--
Best regards,
Vladimir