[certi-dev] HLA_TestsSuite usage

[Eric Noulard]

Hi all,

As I already said thanks to some of you we are
building a growing number of HLA tests programs
which are meant to be used for either conformance
and regression tests suite for CERTI (or may be other RTI).

The tracker task describing the idea is there:
https://savannah.nongnu.org/task/index.php?6909

The HLA_TestsSuite cvs module is there:
http://cvs.savannah.gnu.org/viewvc/applications/HLA_TestsSuite/?root=certi

The idea is build a whole set of programs which may be runned
easily and systematically. Since automated testing of distributed
software is not an easy task we use a combination of CTest and DTest
to be able to run the test suite.

Here is a small explanation on how it is done.
In HLA_TestsSuite/CMakeLists.txt we find statement like:

HLATestsSuite_ADD_TEST(test_Sync
                      FED_FILE test_Sync.fed
                      SOURCE_FILES test_Sync.cc
                      DTEST_SCRIPT dtest_test_Sync.py
                      COMMENTS "Test HLA Synchronization points and Callbacks")

This defined the "test_Sync" programs with specified:
     SOURCE_FILES, the lists of sources files to compile for the test programs
     FED_FILE, the fed file to be used for this test
     DTEST_SCRIPTS, the dtest script to run in order to perform the test

The HLATestsSuite_ADD_TEST CMake macro
will generate appropriate CMake command needed to build
the test executable and add the DTEST_SCRIPT to the lists
of test to be runned by CTest.

CTest is a CMake companion
http://www.cmake.org/Wiki/CMake_Testing_With_CTest
and
DTest is a python module designed for distributed testing
see http://download.savannah.gnu.org/releases/tsp/dtest/what_is_dtest.pdf.

Even if it may seem a bit complicated once installed
using the whole thing is pretty simple:

0) Check out HLA_TestsSuite
1) build HLA_TestsSuite using CMake
2) install dtest (http://download.savannah.gnu.org/releases/tsp/dtest/)
3) eventually update your Python PATH if you did
   not install dtest globally (or you don't have the right to do so)
   export PYTHONPATH=/path/to/dtest
4) make test
   which should gives something like:

$ make test
Running tests...
Start processing tests
Test project /home/eric/workspace/HLA_TestsSuite/build
  1/  3 Testing create_destroy                   Passed
  2/  3 Testing test_Sync                        Passed
  3/  3 Testing test_TRTCCallbacks               Passed

100% tests passed, 0 tests failed out of 3
$

make test invoke CTest in order to run registered test.
Each test is a dtest script.
If a test fails, one can either look at the log files (<testname>.in/err/out)
or relaunch the dtest scripts manually:

$ dtest_test_Sync.py
## Test Sequence <HLA test test_Sync Starts> ...
## Number of DTesters = 3
##   Description:
##   Launch RTIG + two test_Sync federate for testing synchronization
point and callback,...
1..29
ok 1 - HLA test test_Sync Starts.
ok 2 - Barrier <RTIG started> crossed by all <3> registered DTester(s)
ok 3 - First Federate started and has joined federation
ok 4 - Barrier <First Federate waiting other(s) before going on>
crossed by all <2> registered DTester(s)
ok 5 - Other Federate started and has joined federation
ok 6 - Barrier <Other Federate started> crossed by all <2> registered DTester(s)
ok 7 - announceSynchronization received - test_Sync_Other1
ok 8 - Barrier <Sync Sequence starts...> crossed by all <2> registered
DTester(s)
ok 9 - synchronizationPointAchieved sent 1 - test_Sync_Other1
ok 10 - federationSynchronized received 1 - test_Sync_Other1
ok 11 - synchronizationPointAchieved sent 2 - test_Sync_Other1
ok 12 - SynchronizationPointRegisterSucceeded received - test_Sync_First
ok 13 - federationSynchronized received 2 - test_Sync_Other1
ok 14 - announceSynchronization received - test_Sync_First
ok 15 - synchronizationPointAchieved sent 3 - test_Sync_Other1
ok 16 - federationSynchronized received 1 - test_Sync_First
ok 17 - federationSynchronized received 3 - test_Sync_Other1
ok 18 - federationSynchronized received 2 - test_Sync_First
ok 19 - synchronizationPointAchieved sent 4 - test_Sync_Other1
ok 20 - federationSynchronized received 3 - test_Sync_First
ok 21 - federationSynchronized received 4 - test_Sync_Other1
ok 22 - federationSynchronized received 4 - test_Sync_First
ok 23 - synchronizationPointAchieved sent 5 - test_Sync_Other1
ok 24 - federationSynchronized received 5 - test_Sync_Other1
ok 25 - federationSynchronized received 5 - test_Sync_First
ok 26 - Barrier <Sync Sequence end.> crossed by all <2> registered DTester(s)
ok 27 - Barrier <All Federate(s) ended> crossed by all <3> registered DTester(s)
ok 28 - HLA test test_Sync Ends.
ok 29 - No Tester did timeout.
## Test Sequence <HLA test test_Sync Starts>: PASSED.
$

Any feedbacks regarding HLA_TestsSuite and/or DTest is welcome.

-- 
Erk