Re: [lmi] Headless GUI tests

[Greg Chicares]

On 10/16/20 12:52 PM, Greg Chicares wrote:
[...]
> $Xvfb :1 -screen 0 1280x960x24 &                                      
> [1] 3687
> $time DISPLAY=:1 ./gui_test.sh                                       
> NOTE: starting the test suite
> SUCCESS: 21 tests successfully completed.
> NOTE: 4 tests were skipped
> DISPLAY=:1 ./gui_test.sh  27.68s user 9.13s system 36% cpu 1:39.78 total
> $jobs -l                                                                    
> [1]  + 3687 running    Xvfb :1 -screen 0 1280x960x24
> $kill 3687
> $
> [1]  + done       Xvfb :1 -screen 0 1280x960x24

In a freshly-generated 'bookworm' chroot, running lmi's automated
GUI test under 'xvfb' works, on my own machine at least. It's
slower than running the GUI test directly:

/opt/lmi/src/lmi[0]$time /opt/lmi/src/lmi/gui_test.sh
NOTE: starting the test suite
SUCCESS: 21 tests successfully completed.
NOTE: 4 tests were skipped
/opt/lmi/src/lmi/gui_test.sh  23.23s user 8.42s system 70% cpu 45.147 total

as opposed to running it via 'xvfb-run':

/opt/lmi/src/lmi[0]$time xvfb-run ./gui_test.sh
NOTE: starting the test suite
SUCCESS: 21 tests successfully completed.
NOTE: 4 tests were skipped
xvfb-run ./gui_test.sh  26.34s user 11.18s system 36% cpu 1:42.36 total
X connection to :99 broken (explicit kill or server shutdown).
X connection to :99 broken (explicit kill or server shutdown).

...but that's okay, as discussed previously. And the
"connection broken" messages are just a nuisance that
can safely be ignored.

However, in my old 'bullseye' chroot, the GUI test works
only if I run it directly, without 'xvfb':

/opt/lmi/bin[0]$time /opt/lmi/src/lmi/gui_test.sh
NOTE: starting the test suite
SUCCESS: 21 tests successfully completed.
NOTE: 4 tests were skipped
/opt/lmi/src/lmi/gui_test.sh  21.36s user 8.39s system 70% cpu 42.401 total

...but certain tests fail if I run it via 'xvfb-run':

/opt/lmi/bin[0]$time xvfb-run /opt/lmi/src/lmi/gui_test.sh
NOTE: starting the test suite
paste_census: ERROR (Assertion '(lmi::ssize(grid_window->GetSelectedRows())) == 
(1)' failed (expected 1 vs observed 0). [wx_test_paste_census.cpp : 269] )
pdf_census: ERROR (Assertion 'composite_pdf.exists()' failed. 
[wx_test_pdf_create.cpp : 160] )
pdf_illustration: ERROR (Assertion failure: Expected edit cell dialog to test 
$1,000,000,000 scaling was not shown. [file /opt/lmi/src/lmi/wx_test_new.hpp, 
line 62, in do_new_illustration()].)
validate_output_census: ERROR (Assertion failure: Expected case defaults dialog 
was not shown. [file /opt/lmi/src/lmi/wx_test_validate_output.cpp, line 150, in 
init_test_census()].)
validate_output_illustration: ERROR (Assertion failure: Expected illustration 
properties dialog was not shown. [file /opt/lmi/src/lmi/wx_test_new.hpp, line 
62, in do_new_illustration()].)
validate_output_mec: ERROR (Assertion failure: Expected new MEC parameters 
dialog was not shown. [file /opt/lmi/src/lmi/wx_test_validate_output.cpp, line 
573, in run()].)
FAILURE: 6 out of 25 tests failed.
NOTE: 4 tests were skipped
X connection to :99 broken (explicit kill or server shutdown).
X connection to :99 broken (explicit kill or server shutdown).
xvfb-run /opt/lmi/src/lmi/gui_test.sh  29.26s user 31.13s system 0% cpu 
3:35:14.96 total

[the "total" time is correct: it ran that long, in a forgotten
session, until I eventually noticed it and killed it]

...and the same tests fail if I emulate 'xvfb-run' manually,
as in the year-old email quoted above:

/opt/lmi/bin[0]$Xvfb :1 -screen 0 1280x960x24 &
[1] 355948
/opt/lmi/bin[0]$_XSERVTransmkdir: Owner of /tmp/.X11-unix should be set to root

/opt/lmi/bin[0]$time DISPLAY=:1 /opt/lmi/src/lmi/gui_test.sh              
NOTE: starting the test suite
paste_census: ERROR (Assertion '(lmi::ssize(grid_window->GetSelectedRows())) == 
(1)' failed (expected 1 vs observed 0). [wx_test_paste_census.cpp : 269] )
pdf_census: ERROR (Assertion 'composite_pdf.exists()' failed. 
[wx_test_pdf_create.cpp : 160] )
pdf_illustration: ERROR (Assertion failure: Expected edit cell dialog to test 
$1,000,000,000 scaling was not shown. [file /opt/lmi/src/lmi/wx_test_new.hpp, 
line 62, in do_new_illustration()].)
validate_output_census: ERROR (Assertion failure: Expected case defaults dialog 
was not shown. [file /opt/lmi/src/lmi/wx_test_validate_output.cpp, line 150, in 
init_test_census()].)
validate_output_illustration: ERROR (Assertion failure: Expected illustration 
properties dialog was not shown. [file /opt/lmi/src/lmi/wx_test_new.hpp, line 
62, in do_new_illustration()].)
validate_output_mec: ERROR (Assertion failure: Expected new MEC parameters 
dialog was not shown. [file /opt/lmi/src/lmi/wx_test_validate_output.cpp, line 
573, in run()].)
FAILURE: 6 out of 25 tests failed.
NOTE: 4 tests were skipped
^C
DISPLAY=:1 /opt/lmi/src/lmi/gui_test.sh  0.06s user 0.05s system 0% cpu 2:51.31 
total
/opt/lmi/bin[130]$jobs -l
[1]  + 355948 running    Xvfb :1 -screen 0 1280x960x24
/opt/lmi/bin[0]$kill 355948
/opt/lmi/bin[0]$
[1]  + done       Xvfb :1 -screen 0 1280x960x24

As can be seen from the "^C" above, the command did not
finish normally, so I killed it manually.

Thus, in the old 'bullseye' chroot, with 'xvfb', some tests fail,
and the process has to be killed manually, which is less than ideal
for an automated test.

If I run only one sub-test that happens not to fail, then everything
behaves as hoped in the 'bullseye' chroot:

/opt/lmi/bin[0]$xvfb-run wine ./wx_test --ash_nazg  --data_path=/opt/lmi/data 
--test about_dialog_version
NOTE: starting the test suite
about_dialog_version: started
About dialog version string is '20211025T2123Z'.
time=2096ms (for about_dialog_version)
about_dialog_version: ok
time=2100ms (for all tests)
SUCCESS: 1 test successfully completed.
X connection to :99 broken (explicit kill or server shutdown).                  
             

Vadim, do you have any idea how I can debug this further?