[Tsp-devel] dtest: a distributed test framework

[Frederik Deweerdt]

Hi,

Following the discussion on the need of a distributed test system, I
remembered that I had a python piece of code that just does that. See
below for more details on how it works.

A few things first:
- The code is experimental, it may do horrible things to your box
- Scripts are deployed by ssh, your pubkey must be in the target machine
authorized_keys (test it first on localhost as the TARGET=localhost in
launch.sh shows)
- If you test at home, change the USER variable in the launch.sh script,
this is the target machine user the scripts will run with.
- At the moment the client/server run on the same machine. This is not
a limitation of the architecture, it's hardcoded because I laked time.
- You probably need to run "pytap_svn/pytap/trunk/setup.py install" as
root to get the pyTAP module installed in your system
- I'm no python expert, the code may seem awful to some ;)

How it works:
=============

Step 1:
Write tests scripts as python scripts, write their parameters in a _spec file.
The test scripts will get a TestHelper providing two methods:
ok(successful?, "message")
barrier(identifier)

The ok method reports whenever a test has been successful or not, and
an optional explanation message.

The barrier method is a synchronisation method. Let's see that with the
following example involving a test client and a test server.

Client                  Server
Run

                       Run

barrier("init")         //Do complex
...                     //queries on
//blocks                //a database
...                     //and other
                       //fancy stuff

                       barrier("init")
Continue run            Continue run

The example above shows that both the server or the client can cross the
"init" barrier if all the testers involved have called it.

Step 2:

Once the scripts are written, we run make to use cxFreeze to generate
binaries from the python scripts.  This step is important, because it
allows to deploy the python scripts and whatever their dependencies
WITHOUT HAVING EVEN PYTHON INSTALLED in the target machines.

Step 3:
Launch the tests. We use CTest to run the launch.sh script. The launch.sh
script deploys the generated python binary tests to the target machines
using ssh.

Step 4:
The tests are reported using Test Anything Protocol, which is basically a text
convention allowing to report OK or KO for a given test. See
http://en.wikipedia.org/wiki/Test_Anything_Protocol for more details.
We've taught CTest about TAP, so we're able to have this nice report:

$ make test
Running tests...
Start processing tests
Test project /home/def/Prog/dtest/aze/dtest
 1/  1 Testing test_001                         Passed

100% tests passed, 0 tests failed out of 1

Tree structure explained
========================
.
|-- CMakeLists.txt                      #Handles dependecies between
tests and generated tarballs, CTest commands
|-- cx_Freeze-3.0.3                     #Utility to convert the test
scripts into executables
|-- dist                                #Empty at first, after 'make'
contains the generated executables and .tar files ready to deploy
|   `-- test_001                        #A test
|       |-- client
|       |-- client_spec
|       |-- server
|       |-- server_spec
|       |-- test_001__client.tar        #Generated test for client
|       `-- test_001__server.tar        #Generated test for server
|-- launch.sh                           #Shell script, deploys and
runs the tests (run by 'make test')
|-- src
|   |-- test_engine.py                  #The test engine runs on the
localhost, gathers tests results
|   `-- test_helper.py                  #TestHelper class, provides
the ok() and barrier() methods
`-- tests                               #Contains the sources of the tests
   `-- test_001
       |-- client.py                   #A test client, depending on
the spec file, multiple clients may be run
       |-- client_spec                 #Describes how to launch the test
       |-- server.py                   #A test server
       `-- server_spec                 #Describes how to launch the test



Comments are welcomed.

Regards,
Frederik

dtest.tgz
Description: GNU Zip compressed data