[lmi] Automated GUI testing, revisited

[Greg Chicares]

Now that we have an automated GUI test, I'm starting this new thread
to ask some questions about what it's intended to do, and to discuss
where we should go from here.

I created a 'wx_test.conf' from an old off-list message, and hoped
this would cause more tests to pass, but I still get:
  FAILURE: 11 out of 24 tests failed
(full output below[0]). I understand that the 'benchmark_census'
won't pass until I get copies of the (proprietary) files it uses,
but I'd like to find a way to get more of the other twelve failing
tests to pass.

Randomly picking 'extract', I looked at 'wx_test_extract.cpp'.
I'm pretty sure I understand this [simulate 'Ctrl-O']:
    ui.Char('o', wxMOD_CONTROL);    // "File|Open"
And I think the following code forms the name of the file to open
by grabbing <default_input_filename> and replacing everything after
the last '/' by "ExtractV6.ill":

    wxFileName fn(configurable_settings::instance().default_input_filename());
    fn.SetFullName("ExtractV6.ill");
    wxTEST_DIALOG
        (wxYield()
         ,wxExpectModal<wxFileDialog>(fn.GetFullPath())
         ,wxExpectModal<wxMessageDialog>(wxOK)  // Dismiss first warning.
         ,wxExpectModal<wxMessageDialog>(wxOK)  // And the second one.
         ,wxExpectDismissableModal<MvcController>(wxID_OK)  // Accept defaults.
        );

But, if I've understood that correctly, given that I have
  <default_input_filename>C:/etc/opt/lmi/default.ill</default_input_filename>
then this should attempt to open a file named:
  C:/etc/opt/lmi/ExtractV6.ill
Am I supposed to have such a file? If so, then I'll have to ask
my USA coworkers to give me a copy of all the files I need.

I have to question why any input file, other than the default
input file itself, would reside in /etc/ . Maybe I just haven't
understood the specifications, but this location seems strange.

Vadim, could I ask you to add comment blocks (e.g, above the
principal function in each 'wx_test*.cpp' file) that explain what
each test is supposed to do? To make this chore practical, I'd
suggest you just cut and paste from the specifications--and to
avoid exposing any proprietary information on the mailing list,
you can just send a patch to my personal account, and I'll remove
anything proprietary before committing.

---------

[0] "full output below":

00:36:58: Starting automatic tests:
00:36:58:     about_dialog_version: ok (14ms)
00:36:58:     benchmark_census: ERROR (Assertion failure: A message box "Sorry, 
could not open this file." was shown unexpectedly. [file 
/opt/lmi/local/include/wx-3.1/wx/testing.h, line 315, in ReportFailure()].)
00:37:02:     calculation_summary: ok (4671ms)
00:37:02:     configurable_settings: ERROR (Assertion 
'fs::exists("/etc/opt/lmi/configurable_settings.xml")' failed. [file 
/lmi/src/lmi/wx_test_config_settings.cpp, line 37] )
00:37:02:     create_open_census: ok (134ms)
00:37:03:     create_open_database: ok (908ms)
00:37:04:     create_open_gpt: ok (322ms)
00:37:05:     create_open_illustration: ok (944ms)
00:37:05:     create_open_mec: ok (224ms)
00:37:05:     create_open_policy: ok (162ms)
00:37:05:     create_open_rounding: ok (271ms)
00:37:06:     create_open_strata: ok (454ms)
00:37:06:     create_open_text: ok (55ms)
00:37:06:     default_input: ERROR (Assertion '(effective_date) == 
(first_of_month)' failed (expected 2456963 vs observed 2456971). [file 
/lmi/src/lmi/wx_test_default_input.cpp, line 48] )
00:37:06:     default_update: ERROR (Assertion failure: A message box "Unable 
to read file 'C:/etc/opt/lmi/default.ill'. [file 
/lmi/src/lmi/illustration_document.cpp, line 110] " was shown unexpectedly, 
expected wxDialog. [file /opt/lmi/local/include/wx-3.1/wx/testing.h, line 315, 
in ReportFailure()].)
00:37:06:     expiry_dates: ERROR (Assertion '(end) == (end_of_month)' failed 
(expected 2450480 vs observed 2458849). [file 
/lmi/src/lmi/wx_test_expiry_dates.cpp, line 77] )
00:37:06:     extract: ERROR (Assertion failure: Expected wxMessageDialog 
dialog was not shown. [file /opt/lmi/local/include/wx-3.1/wx/testing.h, line 
315, in ReportFailure()].)
00:37:06:     input_sequences: ERROR (Assertion failure: A message box "Sorry, 
could not open this file." was shown unexpectedly. [file 
/opt/lmi/local/include/wx-3.1/wx/testing.h, line 315, in ReportFailure()].)
00:37:06:     input_validation: ERROR (Assertion failure: A message box "Sorry, 
could not open this file." was shown unexpectedly. [file 
/opt/lmi/local/include/wx-3.1/wx/testing.h, line 315, in ReportFailure()].)
00:37:07:     paste_census: ok (1016ms)
00:37:07:     pdf_census: ERROR (Assertion 'fs::exists(composite_pdf_path)' 
failed. [file /lmi/src/lmi/wx_test_pdf_create.cpp, line 135] )
00:37:09:     pdf_illustration: ok (2211ms)
00:37:10:     validate_output_illustration: ERROR (Assertion failure: Expected 
wxDialog dialog was not shown. [file 
/opt/lmi/local/include/wx-3.1/wx/testing.h, line 315, in ReportFailure()].)
00:37:11:     validate_output_mec: ERROR (Assertion failure: A message box 
"Sorry, could not open this file." was shown unexpectedly, expected wxDialog. 
[file /opt/lmi/local/include/wx-3.1/wx/testing.h, line 315, in 
ReportFailure()].)
00:37:11: FAILURE: 11 out of 24 tests failed.