RE: [certi-dev] HLA_TestsSuite usage / dtest problems

[Gotthard,Petr]

Eric,
I tried the test suite, but faced some problems...

PROBLEM #0: The CERTI's cmake doesn't detect dtest, so cmake says OK
even if dtest is not ready. For me it would be better if CERTI's cmake
calls dtest-autotest (or whatever else) to verify the test environment
is OK.

So, at first I realized I'm missing dtest. ;-) After installing all four
dtest external dependencies I have also installed dtest.
PROBLEM #1: The dtest README.txt says it can be installed by typing
'python setup.py'. At least on my installation I had to type 'python
setup.py install' (install was missing from the dtest documentation).

PROBLEM #2: When I start make test in certi I get failure for all tests,
so I tried dtest-autotest (dtest-0.5) and got the following

address@hidden external]# python --version
Python 2.5.1
address@hidden external]# dtest-autotest
# OK not adding the run step: <function myInvalidCustomStep at
0xb7bba17c>
## Since it raises :
StepFirstArgNotDTesterNorSelfException('args[0]=bla',)
Traceback (most recent call last):
  File "/usr/bin/dtest-autotest", line 100, in <module>
    myDTestMaster =
dtest.DTestMaster("DTest-Autotest",description="DTest
autotests sequence not using sesssion")
  File "/usr/lib/python2.5/site-packages/dtest/dtestmaster.py", line
134, in
__init__
    self.traceManager.newSequence(self)
  File "/usr/lib/python2.5/site-packages/dtest/trace_manager.py", line
70, in
newSequence
    handler.newSequence(dtestmaster)
  File "/usr/lib/python2.5/site-packages/dtest/TAP_trace_handler.py",
line 42,
in newSequence
    self.builder      = TAP.Builder.create(out=self.output)
TypeError: create() got an unexpected keyword argument 'out'

Any idea what could be wrong? I'm not a python expert, so unfortunatelly
I will need some more help with this.


Petr

-----Original Message-----
From: address@hidden
[mailto:address@hidden On
Behalf Of Eric Noulard
Sent: Wednesday, June 25, 2008 11:52 AM
To: CERTI development discussions
Subject: [certi-dev] HLA_TestsSuite usage

Hi all,

As I already said thanks to some of you we are building a growing number
of HLA tests programs which are meant to be used for either conformance
and regression tests suite for CERTI (or may be other RTI).

The tracker task describing the idea is there:
https://savannah.nongnu.org/task/index.php?6909

The HLA_TestsSuite cvs module is there:
http://cvs.savannah.gnu.org/viewvc/applications/HLA_TestsSuite/?root=cer
ti

The idea is build a whole set of programs which may be runned easily and
systematically. Since automated testing of distributed software is not
an easy task we use a combination of CTest and DTest to be able to run
the test suite.

Here is a small explanation on how it is done.
In HLA_TestsSuite/CMakeLists.txt we find statement like:

HLATestsSuite_ADD_TEST(test_Sync
                      FED_FILE test_Sync.fed
                      SOURCE_FILES test_Sync.cc
                      DTEST_SCRIPT dtest_test_Sync.py
                      COMMENTS "Test HLA Synchronization points and
Callbacks")

This defined the "test_Sync" programs with specified:
     SOURCE_FILES, the lists of sources files to compile for the test
programs
     FED_FILE, the fed file to be used for this test
     DTEST_SCRIPTS, the dtest script to run in order to perform the test

The HLATestsSuite_ADD_TEST CMake macro
will generate appropriate CMake command needed to build the test
executable and add the DTEST_SCRIPT to the lists of test to be runned by
CTest.

CTest is a CMake companion
http://www.cmake.org/Wiki/CMake_Testing_With_CTest
and
DTest is a python module designed for distributed testing see
http://download.savannah.gnu.org/releases/tsp/dtest/what_is_dtest.pdf.

Even if it may seem a bit complicated once installed using the whole
thing is pretty simple:

0) Check out HLA_TestsSuite
1) build HLA_TestsSuite using CMake
2) install dtest (http://download.savannah.gnu.org/releases/tsp/dtest/)
3) eventually update your Python PATH if you did
   not install dtest globally (or you don't have the right to do so)
   export PYTHONPATH=/path/to/dtest
4) make test
   which should gives something like:

$ make test
Running tests...
Start processing tests
Test project /home/eric/workspace/HLA_TestsSuite/build
  1/  3 Testing create_destroy                   Passed
  2/  3 Testing test_Sync                        Passed
  3/  3 Testing test_TRTCCallbacks               Passed

100% tests passed, 0 tests failed out of 3 $

make test invoke CTest in order to run registered test.
Each test is a dtest script.
If a test fails, one can either look at the log files
(<testname>.in/err/out) or relaunch the dtest scripts manually:

$ dtest_test_Sync.py
## Test Sequence <HLA test test_Sync Starts> ...
## Number of DTesters = 3
##   Description:
##   Launch RTIG + two test_Sync federate for testing synchronization
point and callback,...
1..29
ok 1 - HLA test test_Sync Starts.
ok 2 - Barrier <RTIG started> crossed by all <3> registered DTester(s)
ok 3 - First Federate started and has joined federation ok 4 - Barrier
<First Federate waiting other(s) before going on> crossed by all <2>
registered DTester(s) ok 5 - Other Federate started and has joined
federation ok 6 - Barrier <Other Federate started> crossed by all <2>
registered DTester(s) ok 7 - announceSynchronization received -
test_Sync_Other1 ok 8 - Barrier <Sync Sequence starts...> crossed by all
<2> registered
DTester(s)
ok 9 - synchronizationPointAchieved sent 1 - test_Sync_Other1 ok 10 -
federationSynchronized received 1 - test_Sync_Other1 ok 11 -
synchronizationPointAchieved sent 2 - test_Sync_Other1 ok 12 -
SynchronizationPointRegisterSucceeded received - test_Sync_First ok 13 -
federationSynchronized received 2 - test_Sync_Other1 ok 14 -
announceSynchronization received - test_Sync_First ok 15 -
synchronizationPointAchieved sent 3 - test_Sync_Other1 ok 16 -
federationSynchronized received 1 - test_Sync_First ok 17 -
federationSynchronized received 3 - test_Sync_Other1 ok 18 -
federationSynchronized received 2 - test_Sync_First ok 19 -
synchronizationPointAchieved sent 4 - test_Sync_Other1 ok 20 -
federationSynchronized received 3 - test_Sync_First ok 21 -
federationSynchronized received 4 - test_Sync_Other1 ok 22 -
federationSynchronized received 4 - test_Sync_First ok 23 -
synchronizationPointAchieved sent 5 - test_Sync_Other1 ok 24 -
federationSynchronized received 5 - test_Sync_Other1 ok 25 -
federationSynchronized received 5 - test_Sync_First ok 26 - Barrier
<Sync Sequence end.> crossed by all <2> registered DTester(s) ok 27 -
Barrier <All Federate(s) ended> crossed by all <3> registered DTester(s)
ok 28 - HLA test test_Sync Ends.
ok 29 - No Tester did timeout.
## Test Sequence <HLA test test_Sync Starts>: PASSED.
$

Any feedbacks regarding HLA_TestsSuite and/or DTest is welcome.

--
Erk


-- 
CERTI-Devel mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/certi-devel