Re: [lmi] Automated GUI testing, revisited

[Greg Chicares]

On 2014-11-09 01:21Z, Greg Chicares wrote:
> Now that we have an automated GUI test, I'm starting this new thread
> to ask some questions about what it's intended to do, and to discuss
> where we should go from here.

If I follow the instructions in 'INSTALL' and then run 'wx_test',
then I tend to think that all the tests it runs should pass,
unless there was a serious problem building it.

It seems that we've gone beyond the idea of automating GUI tests,
and added numerous tests that I guess either use proprietary
products, or validate binary distributions. That may be very
important and useful, but if we're combining diverse goals, then
I'd like to separate them, and by default run only the tests that
should pass in a default installation (following 'INSTALL').

I know we have command-line options available (quoting from an
off-list email without prior permission because I feel certain
it's all right, but tell me if I've overstepped my bounds):

        $ ./lmi_wx_test.exe
        # Runs all tests
        $ ./lmi_wx_test.exe -t about_dialog_version
        # Runs only the test #4 from the specification
        $ ./lmi_wx_test.exe --test -expiry_dates
        # Runs all tests except for the test #3 in the specification
        $ ./lmi_wx_test.exe -t default_input -t configurable_settings
        # Runs only the two specified tests

and I'm asking for something in addition to that:
  [no option specified] Run exactly the tests that must always pass,
    unless something is unequivocally wrong.
  --distribution Also run tests that validate a binary distribution:
    for instance, expiry date = first of month.
  --extra [that name could use improvement] Also run tests that
    depend on proprietary files or products.
I guess I wouldn't complain about an '--everything' option that
enables all of the above. (If there are other categories beyond
those, this is the time to speak up.)

It's okay with me if we treat '--test=' as forbidden when one of
the additional options above is specified.

I don't know which tests fall into which category. I also don't
know whether any tests span multiple categories. I'll have to ask
someone else in the USA to answer those questions.

In default mode--with no command-line options--is 'wx_test.conf'
even useful? It specifies proprietary 'benchmark_census' files;
if that's its sole purpose, then in default mode it shouldn't
even need to exist.

BTW, './wx_test.exe --help' doesn't seem to DTRT. I think it
should mention '--help' and '--data_path', but not the other two
options it shows now; and '--test' and '--list' should be added
along with any new options we add. I'm guessing that this won't
be too hard to do by analogy with Skeleton::ProcessCommandLine(),
but tell me if I'm wrong.