Re: [lmi] Preserving configurable settings after GUI test

[Vadim Zeitlin]

On Thu, 19 Jul 2018 16:37:56 +0000 Greg Chicares <address@hidden> wrote:

GC> On 2018-07-03 00:01, Vadim Zeitlin wrote:
GC> > 
GC> >  There is a long standing TODO item about preserving the contents of
GC> > configurable_settings.xml file after running the GUI unit test
GC> 
GC> Here's a separate, brand-new issue: apparently a race condition when
GC> the GUI test is run concurrently with other tests (with origin/master,
GC> without the patch you proposed):
[...]
GC> This is the first time I've seen this problem, but it can
GC> occur at any time: lmi binaries require that xml file, but they
GC> might try to read it exactly when the GUI test is in the process
GC> of modifying it.

 Yes, and I don't really see any good way around it. One possibility would
be to run all the tests except the one modifying the configuration file
(there is no convenient way to exclude a test from the test harness
command line currently, but this would be easy to add) concurrently with
the nychthemeral test and then run this one test separately after the
nychthemeral test completion. But this adds another manual step and is
going to be difficult to remember, especially because things will sometimes
work even if you forget to exclude the problematic test from the first run.

GC> That's a shame, because it's very convenient to run all tests this
GC> way. The GUI test is by its nature fragile in the sense that it
GC> needs complete control over the console, so I run it manually; and
GC> it's slow enough that I like to run other tests in the background
GC> at the same time.
GC> 
GC> Maybe the answer is to virtualize the xml file, using, say,
GC>   /opt/lmi/wherever/gui_test/configurable_settings.xml
GC> for the GUI test only.

 This is easily doable, of course, but I'd prefer to run the tests in the
same environment as will be really used in production, if possible. Of
course, it seems rather unlikely that the file location can change the test
outcome (and e.g. make it pass while the same operation in the released
version would fail), but who knows how things could change in the future.

GC> IIRC, I recently provided read-only access to its default pathname; for
GC> this special purpose, it may be better to provide a different pathname.

 This will configurable_settings logic more complicated too, which seems to
be undesirable.


 I admit I don't have any obviously great solution to this problem, but the
best one might be to provide a special "ephemeral" mode in which
configurable_settings class would only save changes done to it in memory,
instead of writing them to the disk file. If we used this mode in the
tests, it would ensure that they never modify any disk files but can still
verify that configurable settings are updated as expected.

 What do you think of this approach? Should I propose a patch implementing
it? Of course, this would invalidate https://github.com/vadz/lmi/pull/88/
so if you plan on applying that PR, please don't do it if you think we
should do this.

 Also, my proposed "ephemeral mode" is, of course, just a special case of a
more general idea of mocking up the file system during testing. In
principle, if you like this idea, we could add support for mocking to all
tests, which would also automatically take care of leaving various output
files lying around after the test termination. I probably (i.e. unless you
tell me not to do it) will still implement manual cleanup for them to solve
the problem for you in the near term but if the idea of virtualizing file
system looks appealing to you, it would provide a more elegant way of
ensuring that no such files are left completely effortlessly.

 Please let me know what do you think,
VZ