Re: [lmi] Headless GUI tests

[Vadim Zeitlin]

On Fri, 16 Oct 2020 12:52:47 +0000 Greg Chicares <gchicares@sbcglobal.net> 
wrote:

GC> Using Xvfb instead still takes 100 seconds instead of 40:
GC> 
GC> $Xvfb :1 -screen 0 1280x960x24 &                                      
GC> [1] 3687
GC> $time DISPLAY=:1 ./gui_test.sh                                       
GC> NOTE: starting the test suite
GC> SUCCESS: 21 tests successfully completed.
GC> NOTE: 4 tests were skipped
GC> DISPLAY=:1 ./gui_test.sh  27.68s user 9.13s system 36% cpu 1:39.78 total
GC> $jobs -l                                                                    
GC> [1]  + 3687 running    Xvfb :1 -screen 0 1280x960x24
GC> $kill 3687
GC> $
GC> [1]  + done       Xvfb :1 -screen 0 1280x960x24
GC> 
GC> so the one-minute delay isn't attributable to the xvfb-run script.

 Hello again,

 We've been able to confirm that we also see the same ~1 minute delay and
it's indeed not related to using xvfb-run rather than Xvfb itself directly.
However there doesn't seem to be anything especially worrisome about this,
it just seems that Xvfb is less efficient than the actual production X
server -- which is not really surprising. In fact, potentially more
surprising thing is that, looking at the time of executing individual tests
(all times are in seconds):

        Test name                    | Screen |   Xvfb
        -----------------------------|--------|--------
        about_dialog_version         |    6.2 |   10.1
        benchmark_census             |    6.3 |    5.8
        calculation_summary          |   26.5 |   29.3
        configurable_settings        |    4.4 |    6.0
        create_open_census           |    4.4 |    5.8
        create_open_database         |    7.7 |    7.5
        create_open_gpt              |    8.4 |    7.5
        create_open_illustration     |   14.3 |   13.8
        create_open_mec              |    8.0 |    7.1
        create_open_policy           |    8.6 |    7.7
        create_open_rounding         |   10.9 |    8.7
        create_open_strata           |    5.5 |    8.4
        create_open_text             |    3.8 |    6.4
        default_input                |    7.0 |    5.1
        default_update               |    3.4 |    7.6
        expiry_dates                 |    4.2 |    5.9
        input_sequences              |   77.0 |   96.5
        input_validation             |    5.8 |    6.1
        log_error                    |    8.7 |    5.4
        paste_census                 |   17.5 |   21.2
        pdf_census                   |    7.6 |    8.9
        pdf_illustration             |   11.6 |   13.7
        validate_output_census       |   24.3 |   29.2
        validate_output_illustration |   11.2 |   15.5
        validate_output_mec          |    7.2 |    8.9

you can see that some tests run _faster_ under Xvfb than under the real X
server. However this could also be explained by e.g. Xvfb not implementing
some features that the production server implements, so I don't think it's
a huge mystery neither. If you're really curious, we could try to find out
which X calls take more/less time, but I don't think it's really worth
doing it, is it?

 IMO the important thing is that this 1 minute difference doesn't
correspond to some "sleep 1m" (sorry for a GNU-ism) somewhere, but just
happens to be the sum of the differences of the individual tests.

 If anything, I'd rather look into input_sequences test to see why does it
take so long and check if we could optimize it without losing any test
coverage. Do you think this could be worth doing?


GC> > GC> $time xvfb-run ./gui_test.sh                                    
GC> > GC> NOTE: starting the test suite
GC> > GC> Abnormal-termination handler called. Please report this problem.
GC> > 
GC> >  Should I/we look into this or are you already looking or have already
GC> > found the source of this problem?
GC> 
GC> TL;DR: Yes please.

 I'm not sure if it's the same crash that you're seeing, so I'd like to
ask: does /opt/lmi/gui_test exist on your machine? If it does, this must
be something else and I don't know what it is yet. However if it doesn't,
then we do observe the behaviour above and I understand why does it happen
and just didn't have to make a PR fixing it yet.


GC> I had hoped that I could fix this little nuisance easily, and then
GC> invoke wx_test from nychthemeral_test, as I have wished for so long;
GC> and then turn to some pressing work. But my ambition to clean up
GC> nuisances exceeded my expertise in this and a couple other cases,
GC> so I'd be very glad if you'd look into this.

 FWIW the CI builds on GitHub can run all the tests, including the GUI
ones, successfully for MSW binaries. Unfortunately there are some
non-trivial problems with the GUI tests and wxGTK, that we're still working
on, so it might take slightly longer before we can run all the tests for
the native builds too.


 To summarize the various issues:

- I think there is no real problem with the tests performance under Xvfb,
  but please let me know if you disagree.

- We've debugged a problem due to calling wxMessageBox() too early in the
  tests and will fix it soon, but if the crash you've seen is not due to
  it (the simplest way to see it would be to run it under gdb and look at
  the stack when the exception is thrown, i.e. do "catch throw" and "bt"
  when it's triggered in gdb), then we still don't know what is it.

- We're still working on making the GUI tests work with wxGTK, I think it's
  worth pursuing even, but I realize that these binaries are not used in
  production currently, so, again, please let me know if you disagree.

 Thanks in advance,
VZ

pgpEL1h6ZDLvP.pgp
Description: PGP signature