Re: [lmi] BOOST_TEST needed for automated GUI testing?

[Greg Chicares]

On 2014-03-17 13:05Z, Václav Slavík wrote:
> 
> On 16 Mar 2014, at 17:53, Greg Chicares <address@hidden> wrote:
>> Is it really important to use lmi's BOOST_TEST macro in
>> 'main_wx_test.cpp', or would any other condition-testing
>> macro do as well?
> 
> It isn’t, I did it for consistency with other tests - it doesn’t
> make much sense to use two different test frameworks, does it?

The existing test framework has evolved for small, standalone
unit tests. In particular, it implements a non-GUI main() in a
uniform way that encapsulates initialization and termination
of the testing framework. And that main() has its own set of
command-line options that are not appropriate for GUI testing.
Of course, we could decouple that main() implementation...

> Would it make sense to either
> 
> a) decouple test_tools.hpp from cpp_main.cpp, require the latter to
> be explicitly included and implement a similar, but not identical,
> version in main_wx_test.cpp; or
> 
> b) have some BOOST_TEST_ALLOW_UNRECOGNIZED_OPTIONS symbol, #define
> it in main_wx_test.cpp before including test_tools.hpp and modify
> cpp_main.cpp to report options errors only if this symbol is not defined
> 
> instead? Then the same set of macros could be used without this conflict.

Yes, we could do that--but to what end? AFAICT, BOOST_TEST() is
the only one of those macros that will be useful for GUI testing,
whereas the others are designed for testing C++ exceptions and
numerical calculations, which I doubt we'll find useful in this
context. So we have another couple of options:

c) Clone BOOST_TEST(), and the accompanying framework code that
counts and reports the number of succeeding and failing tests; or

d) Just use LMI_ASSERT() instead of BOOST_TEST(), reasoning that
the extra work of (c) is not worth the corresponding benefit (and
that none of the other macros will prove useful for GUI testing).

I don't remember all the other kinds of tests that will eventually
be wanted, but we don't need to make a final decision today for
the one specimen test that's implemented right now--so, in order
to prevent the problems described below, I'm going to go ahead and
implement (d) for now.

>> I run the commands above, this appears on the console:
>>  C:\opt\lmi\lmi-20140316T1359Z\wx_test.exe: unrecognized option 
>> '--data_path=/opt/lmi/data'
>>  Unrecognized option '--data_path=/opt/lmi/data’.
> 
> Ah, I didn’t notice this in my testing because it scrolled off the window and 
> the test executed.

(That could be prevented with '--accept', but then that option
would be rejected by the GUI command-line parser.)

>> and this messagebox pops up:
>>  wxWidgets Debug Alert
>>  /opt/lmi/local/include/wx-3.0/wx/testing.h(268): \
>>    assert "Assert failure" failed in ReportFailure(): \
>>    Expected wxDialog dialog was not shown.
>>  Do you want to stop the program?
>>  You can also choose [Cancel] to suppress further warnings.
>>  Yes   No   Cancel
> 
> This doesn’t happen here. This isn’t something that could be easily
> explained by compiler differences (and I tested with MinGW too when
> testing my makefile changes anyway), nor do I see any relevant changes
> in the SVN version of the patches.  I suspect it may be something in wx
> that was only fixed after 3.0 was released (I used a more recent snapshot)
> - I’ll rebuild with 3.0 and retest.

That could be interesting to know. However, I think I
guessed wrong here:

>> As far as I can guess, the "unrecognized option" warning
>> interferes with the GUI test, causing it to fail before
>> expect_about_dialog::OnInvoked() is even called.
> 
> That would make some sense if the error was reported via wxLog(), but it 
> isn’t.

Building with not-yet-committed local changes, I observe this:

  /opt/lmi/bin[0]$./wx_test.exe --ash_nazg --data_path=/opt/lmi/data
Everything works as expected.

  /opt/lmi/bin[0]$./wx_test.exe --mellon --data_path=/opt/lmi/data
The above "wxWidgets Debug Alert" appears.

My new hypothesis is that the "Debug Alert" occurs because
the "about" dialog is automatically displayed at startup in
the second case. I.e., it's displayed automatically by the
production system, and therefore by the GUI-test binary too;
so the GUI-test binary's attempt to display it *again* under
program control fails (because a modal dialog is already shown,
blocking menu-command input).