Re: [lmi] Headless GUI tests

[Greg Chicares]

On 11/13/21 12:54 PM, Vadim Zeitlin wrote:
> On Fri, 12 Nov 2021 22:17:59 +0000 Greg Chicares <gchicares@sbcglobal.net> 
> wrote:
> 
> GC> In a freshly-generated 'bookworm' chroot, running lmi's automated
> GC> GUI test under 'xvfb' works, on my own machine at least.
> 
>  So the solution is clearly to update to Bookworm :-/

We plan to do that as of 2022-01-01, for all machines used for lmi
development in the US. (Unless we find something fatally wrong with
'bookworm', which is highly unlikely.)

On my machine, as long as I'm using 'bullseye', I can continue to
run these two commands separately:
  /opt/lmi/src/lmi/nychthemeral_test.sh
  /opt/lmi/src/lmi/gui_test.sh
as has long been my habit before git-pushing any substantive change.
It would be more convenient to add a headless 'gui_test.sh' to
'nychthemeral_test.sh'. However, the extra convenience I would
realize by doing that now instead of on 2022-01-01 wouldn't justify
much effort.

The important problem is on our corporate server. We log in from
msw laptops using PuTTY. We don't have access to the physical
server (which AFAIK might be virtual rather than physical), so we
really want to run headless GUI tests there. (Otherwise, we must
build on the server, move the built files to msw, and perform a
GUI test there manually--and that's really cumbersome.)

Therefore, let me answer your questions here, in case that helps
us to get headless GUI tests working on the server; and I'll
start a separate thread for server issues.

> GC> It's slower than running the GUI test directly:
> 
>  I think we had checked it before and the explanation was that Xvfb is just
> much slower than the regular X server for many operations

Completely agreed.

I had initially expected that Xvfb would make everything faster,
because naively it seems to have less to do than a real X server.
But physical graphics are probably well optimized in hardware,
and emulation in software may very well be slower.

> GC> However, in my old 'bullseye' chroot
[...]
> GC> certain tests fail if I run it via 'xvfb-run':
> GC> 
> GC> /opt/lmi/bin[0]$time xvfb-run /opt/lmi/src/lmi/gui_test.sh
> GC> NOTE: starting the test suite
> GC> paste_census: ERROR (Assertion 
> '(lmi::ssize(grid_window->GetSelectedRows())) == (1)' failed (expected 1 vs 
> observed 0). [wx_test_paste_census.cpp : 269] )
> 
>  Does it always start with this test failure or do different tests fail
> depending on the run/phase on the moon?

Yes, it's reliably reproducible.

>  I don't see anything wrong with this particular test, but we could change
> it, e.g. to perform selection in the grid from the keyboard instead of
> using the mouse, and it might be enough to make it work?

Perhaps: I haven't tried that yet, because I first tried running
standalone tests, and observed something interesting: most of the
individual GUI tests that had failed in a batch run succeed when
run individually. Running the first failing test alone, I see it
fail as above, requiring a ^C to stop it:

/opt/lmi/bin[130]$xvfb-run wine ./wx_test --ash_nazg  --data_path=/opt/lmi/data 
--test paste_census
NOTE: starting the test suite
paste_census: started
paste_census: ERROR (Assertion '(lmi::ssize(grid_window->GetSelectedRows())) == 
(1)' failed (expected 1 vs observed 0). [wx_test_paste_census.cpp : 269] )
time=211ms (for all tests)
FAILURE: 1 out of 1 test failed.
^CX connection to :99 broken (explicit kill or server shutdown).

...but the other tests all run to completion without ^C (i.e.,
'xvfb-run' closes the X connection automatically), and four of
these five tests succeed:

/opt/lmi/bin[130]$xvfb-run wine ./wx_test --ash_nazg  --data_path=/opt/lmi/data 
--test pdf_census
NOTE: starting the test suite
pdf_census: started
time=2771ms (for pdf_census)
pdf_census: ok
time=2773ms (for all tests)
SUCCESS: 1 test successfully completed.
X connection to :99 broken (explicit kill or server shutdown).

/opt/lmi/bin[0]$xvfb-run wine ./wx_test --ash_nazg  --data_path=/opt/lmi/data 
--test pdf_illustration
NOTE: starting the test suite
pdf_illustration: started
time=3779ms (for pdf_illustration)
pdf_illustration: ok
time=3782ms (for all tests)
SUCCESS: 1 test successfully completed.
X connection to :99 broken (explicit kill or server shutdown).

/opt/lmi/bin[0]$xvfb-run wine ./wx_test --ash_nazg  --data_path=/opt/lmi/data 
--test validate_output_census
NOTE: starting the test suite
validate_output_census: started
validate_output_census: ERROR (Assertion '0 <= row && row < 
grid_table_->GetRowsCount()' failed. [census_view.cpp : 1144] )
time=3536ms (for all tests)
FAILURE: 1 out of 1 test failed.
X connection to :99 broken (explicit kill or server shutdown).

/opt/lmi/bin[1]$xvfb-run wine ./wx_test --ash_nazg  --data_path=/opt/lmi/data 
--test validate_output_illustration
NOTE: starting the test suite
validate_output_illustration: started
time=3617ms (for validate_output_illustration)
validate_output_illustration: ok
time=3621ms (for all tests)
SUCCESS: 1 test successfully completed.
X connection to :99 broken (explicit kill or server shutdown).

/opt/lmi/bin[0]$xvfb-run wine ./wx_test --ash_nazg  --data_path=/opt/lmi/data 
--test validate_output_mec
NOTE: starting the test suite
validate_output_mec: started
time=295ms (for validate_output_mec)
validate_output_mec: ok
time=298ms (for all tests)
SUCCESS: 1 test successfully completed.
X connection to :99 broken (explicit kill or server shutdown).

>  I think this happens because some top level window (a dialog which could
> have been shown later than expected, maybe?) must have remained open,
> preventing the program from exiting. It is a bit difficult to check this,
> as you can't look on the Xvfb screen, of course.

I'm looking into 'xwud' in debian's 'x11-apps' package, which
looks like it might actually let us examine the Xvfb screen.

> But I think I should make
> this code more robust in any case, either by extending the destructor of
> ensure_top_window_closed to close all top level windows and not just the
> main one, or by just calling ExitMainLoop() instead of using this class,
> which would seem to be much simpler and more robust and I can't remember
> why I hadn't done it like this from the beginning.

That seems consonant with the results above, which suggest that
the 'paste_census' test is the one that prevents 'wx_test' from
terminating, and perhaps even causes apparent failures in four
later tests that succeed when run independently.

Therefore, yes, would you please provide a patch as you describe above?

>  How critical is fixing this, i.e. how big of a constraint is having to run
> the test in your Bookworm chroot for you?

By New Year's Day I plan to have abandoned my 'bullseye' chroot.
Investigating these problems with 'bullseye' has no inherent value.
But it may shed light on the problems I'm encountering with a
'bookworm' chroot on our corporate server. And it's good in any
case to close all top-level windows as discussed above.