RE: [certi-dev] How to contribute to HLA TestSuite

[Gotthard, Petr]

Hi Eric,

One comment: Would it be possible to prepare a simple dtest template-- two 
"empty" federates that I could copy and update in order to create new a test 
for this suite?

The existing HLA TestSuite tests are quite complex to understand how dtest 
works and how to write custom multi-federate tests. There may be more people 
with a similar problem, so this could help to overcome the "initial resistance" 
of dtest. ;-)


Best Regards,
Petr

-----Original Message-----
From: address@hidden [mailto:address@hidden On Behalf Of Eric Noulard
Sent: Thursday, August 13, 2009 11:24 AM
To: CERTI development discussions
Subject: [certi-dev] How to contribute to HLA TestSuite

Hi All,

Several of you already contribute as bug reporter, features requester
and even patch provider.
That's great, thank you already for that.
For those of you which are interested in other or complementary way
of contributing which will surely enhance CERTI quality, keep reading.

Some time ago we did start the HLA TestSuite which is a big name
for a currently small set of HLA federates which may be used to test
features of any HLA RTI,  including CERTI.

HLA Test Suite is hosted as a CVS sub-module on CERTI project at Savannah:
http://cvs.savannah.gnu.org/viewvc/applications/HLA_TestsSuite/?root=certi

such that you may get the module as you already do with CERTI CVS:

cvs -z3 -d:pserver:address@hidden:/sources/certi co
applications/HLA_TestsSuite
(registered CERTI developers should use authenticated CVS access)

The CVS module contains:

 1) A CMakeLists.txt which is used to compile all the test suite
    (just as you do when compiling CERTI)

 2) A set of tests
      each test consisting in:
        - a set C++ source files,
        - fed files
        - and a python script

The idea of the test suite is to be the base non-regression test for CERTI, but
since all tests consisting in "standard" HLA Federate the HLA TestSuite should
be compiled and runned with other RTI (not done by us yet, but this is
theoretically doable).

The fact is each time we merge-in a patch, a new feature a bug fix etc...
we (CERTI developer) run this Test Suite. This is easy not too long
and it ensures
that we do not introduce regression while fixing bugs.

In fact we should add a test EACH TIME we fix a bug, in order to be sure
the bug do not re-appear afterwards. Sadly, we do not always have time
to do so, thus keep reading.

Adding a test to the test suite using the CMakeLists.txt should be done
by adding lines like the following to the CMakeLists.txt:

HLATestsSuite_ADD_TEST(test_FOMParse
                      FED_FILE TestFed.fed BuggyFed.fed
Hierarchical.fed RPR-FOM.fed aviationsimnetV3.1.xml
                      SOURCE_FILES test_FOMParse.cc
test_FOMParse_cmdline.h test_FOMParse_cmdline.c
                      DTEST_SCRIPT dtest_test_FOMParse.py
                      COMMENTS "A FOM parsing test"
                      )

The CMake macro will try to build the "test_FOMParse" executable using
the files provided
in the SOURCE_FILES section, the FED_FILE is a list of fed/xml files
to be used for the test.
DTEST_SCRIPT is an optional dtest (python) script which is to be used
to automatically run the test.

Ideally each test would end-up being a single executable which is a
RTI federate which should
work with specified fed files.

Since running a test usually requires the launch of several executables like;

1) run rtig
2) run test_whatever -name "Fed1" -f test_whatever.fed
3) run test_whatever -name "Fed2"
4) wait for expected results
5) terminates every process

We did try to automate all thoses steps using DTest which is a small python
framework for doing "distributed testing".
An introduction to dtest is given here
http://download.savannah.gnu.org/releases-noredirect/tsp/dtest/what_is_dtest.pdf

That said the hopefully complete procedure to run HLA TestsSuite with
CERTI is the following:

0) Install CERTI as explained here:
    http://www.nongnu.org/certi/certi_doc/Install/html/index.html

1) Check out HLA_TestsSuite

2) build HLA_TestsSuite using CMake

3) Install HLA_TestsSuite,
   you currently need to install HLA TestsSuite executable in order to makes
   it possible to run (detail explained on request)

4) install dtest (http://download.savannah.gnu.org/releases/tsp/dtest/)

5) eventually update your Python PATH if you did
   not install dtest globally (or you don't have the right to do so)
   export PYTHONPATH=/path/to/dtest

6) make test
    which should gives something like:

$ make test
Running tests...
Start processing tests
Test project /LOCAL/ERIC/workspace/HLA_TestsSuite/build
  1/  6 Testing create_destroy                   Passed
  2/  6 Testing test_Sync                        Passed
  3/  6 Testing test_TRTCCallbacks               Passed
  4/  6 Testing Interactive_Federate             Passed
  5/  6 Testing test_NotificationSwitches        Passed
  6/  6 Testing test_FOMParse                    Passed

100% tests passed, 0 tests failed out of 6
$

Using this "framework" if you propose a new test it will automatically be run
each time we do regression testing on CERTI.

This introduction to DTest and HLA TestsSuite has already been given on the
list some time ago, please check this thread for eventual troubleshooting:
http://lists.nongnu.org/archive/html/certi-devel/2008-06/msg00045.html

Note that the next step concerning CERTI testing is to contribute to the
CERTI and HLA TestsSuite Dashboard using CDash:
http://lists.nongnu.org/archive/html/certi-devel/2008-10/msg00028.html

I think all of this is worth a more structured document, I may try to
write it down
in the future.

Happy CERTI usage & testing.
-- 
Erk
Membre de l'April - < promouvoir et défendre le logiciel libre > -
http://www.april.org


-- 
CERTI-Devel mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/certi-devel