[certi-dev] CERTI message generator

[Eric Noulard]

Hi all,

During some discussion with Andrej which is working on a Java Binding for CERTI
https://savannah.nongnu.org/task/?6905
we did discuss the design and purpose of the CERTI message generator.

We find it useful to share the information with other so here it goes:

CERTI internals are based on messages exchanged between 3 processes:

Federate+LibRTI <--->RTIA  exchange CERTI "Message"
and
RTIA<-->RTIG exchange CERTI "Network Message"

those messages are conveyed through socket (Unix or TCP) and should be
portable, i.e. a Network Message flowing from a machine running Federate+RTIA
to another machine running RTIG. Both host may have different architectures
(32/64 bits, Little/Big Endian byte ordering etc...)

When doing the necessary work for enabling heterogeneous simulation with CERTI
I did start a "Message generator" which digest some message specification and
may produce  C++, Java, Python etc... source code which implement the building
and serialization code for the message.

The generator is a Python script:
http://cvs.savannah.gnu.org/viewvc/certi/scripts/GenerateMessages.py?root=certi&view=markup

And the current CERTI [Network] Message specification are:
http://cvs.savannah.gnu.org/viewvc/certi/scripts/CERTI_NetworkMessage.msg?root=certi&view=markup
http://cvs.savannah.gnu.org/viewvc/certi/scripts/CERTI_Message.msg?root=certi&view=markup

The CERTI message specification language is inspired from Google
protocol buffer but
do not share any code with it. I chose not to use Google protobuf
because I want to
keep CERTI dependencies as low as possible moreover protobuf language
was missing some
badly needed features. Some feature like the "representation" of
"native" message were
contributed by Mathias Frohlich.

Current generator contains a C++ generator and a preliminary Java
generator (work in progress)
As it will be explain later in this mail, the generator consists in
several steps,
the language generation part being the last one.
The generation part is "very" specific due to the "native type"
concept but the previous
steps shoud be fairly generic and may even used by interested projects
outside CERTI.

For the curious here is an overview of the generator way of work:

1) native message like in:

native Handle {
   representation uint32
   language CXX [#include "certi.hh"]
   language Java [import  rti.y.x.Handle]
}

is a way to easily incorporate inside [non-native] message type which
are defined in the native language.
"native" meaning the language of the generator backend and not the
message specification language.

the "representation" tells that this native type may be written/read to/from
a stream as an uint32.

2) [non-native] message :

message M_Create_Federation_Execution : merge Message {
       required string federationName // the federation name
       required string FEDid          // the Federation ID which is in
fact a filename
}

says that the M_Create_Federation_Execution will contains
whetever field a Message (from which it is merging from) contains
plus the described fields. The "merge" may be seen as an inheritance
of data structure
contained in the merge source in the currently defined message.

required "means" that this field will ALWAYS be found in the message stream.
optional "means" the stream will contain:
           - a boolean telling whether if the optional field is there
           - the field itself if the preceding is true
repeated "means" the stream will contain:
           - an uint32 ranging from 0 up to whatever uint32 max value
           - the compact list of "n" values as announced in the previous field.


Each basic type ([u]int8/16/32/64, double, float, string etc...) is encoded
with its fixed size (string is a sequence of char).
So the  "message" construct fully describe the payload of a message.
Now in order to handle heterogeneity each message is encoded in a
"MessageBuffer" (you can have a look at the class with the same name
 in the C++ sources).

A message buffer is built with:

5 "reserved bytes"

byte 0 contains endianity information:  0x01 --> BigEndian
                                                       0x00 --> LittleEndian
byte 1..4 represents the size of the [whole] buffer as an uint32
 (coded in the endianity described in the first byte)

"whole" buffer means including the 5 reserved bytes.

Thus theoretically in order to fully exploit the Generator you'll have to:

A) Hand-Code an equivalent of the C++ MessageBuffer class in order
    to be able to easily encode basic types.

B) Write the Java Generator backend which goes through the "pseudo"
    Abstract Syntax Tree (AST) given by the PLY parser.

   The PLY parser built a "MessageAST" object from the message
specification file. The MessageAST is the root of the AST tree
and then contains [list of] object describing
- package
- factory
- native types
- message types
- enum types

Look at each ASTElement derived classes:
   Package
   Factory
   NativeType
   MessageType
   EnumType

each of those python objects contains the informations you find in the
message specification file.

The checker object is used to verify/ensure properties on the AST.
Then the different Generator (which should preferably be derived from
CodeGenerator)
goes through the AST and generate appropriate source code.

In C++ we generated one C++ class for each Message type
those class inherit from "native" Message class (RTIA case) or
"NetworkMessage" class (RTIG case). This is not mandatory and was used
in order to ease the transition from CERTI legacy code to generated one.

In fact currently NM_Classes.[hh,cc] is 98% generated code
and M_Classes.[hh,cc] files should appear soon with the same objective.

The M_Classes.* files should replace 90% of the ugly hand-written code
now found in Message_R.cc and Message_W.cc.

I hope this may have help you to understand the purpose of the generator idea.
In the end if all language binding use the generator any change/update/addition
of message may be done by updating the spec and regenerating the appropriate
code. Some hand-written code may be needed too but this should be minimal.

For example re-ordering differents field of a message should be done
automagically by regenerating the code.

-- 
Erk
Membre de l'April - « promouvoir et défendre le logiciel libre » -
http://www.april.org