[Discuss-gnuradio] [RFC] PMT succession: experts' opinions on serialization libs

[Marcus Müller]

To avoid the setting in of post-Conference downs:

Dearest SDR Community with the Highest count of "Awesome" in the
Universe,

discussions at GRCon have made it abundantly clear that PMT needs to be
replaced¹.
Main issues with PMT are:

* broken type system,
* low suitability as tool for internal packeted data,
* lack of interoperability with anything but the languages we support
natively,
* platform-dependency of the serialization format,
* bugs.

(to a far lesser degree: performance.)

Having talked to multiple domain experts in networked systems and
heterogenous cooperative platforms, I've come to the conclusion,
furthermore, that in the interest of RPC'ing everything², the structure
of messages exchanged between blocks should be predefined.

In this sense, PMT's (limited) ability do describe itself is of no help
– instead, it introduces ambiguity.

The dominant choice for binary serialization with known schemata is
Proto Buffers; however, they require deserialization of a whole message
if there's substructures before the field you want to access³.

Cap'n Proto[1] (by one of the original authors of protobufs) seems to
be a suitable alternative, and comparing it to its competitors, it
seems to have achieved a significant level of maturity and has at least
one large-scale adopter⁴.

Among its advantages seems to be that operations on the message happen
in-place (i.e. there's actually a compatible C++ struct representation
of the memory layout), which rendered it my favourite library for this
job.

Now, problem: I've never personally worked with Cap'n Proto, or
protobuf, or any modern, non-self-describing serialization library;
I've got minimal experience with MessagePack, but as said, that's not
probably going to make it due to lack of schema.

So, here I am, calling for your experience! Hurl ins(igh|ul)ts at me,
preferable here on this list.

Best regards,
Marcus


[1] https://capnproto.org/index.html

¹This doesn't come as a surprise – it's just a good point in time to
point this out.
²You've probably noticed that we're working towards a less machine-
dependent, more portable, distributable computational model
³This can become problematic for messages with large data blobs in
between, or very complex protocol headers
⁴CloudFlare seems to be using Cap'n Proto for Web workers, for example.
However, it's but at revision 0.7, which indeed implies a bit of
headache – other tools that never reached 1.0 that we rely on include
Thrift, and that's really not a club that you'd want to be in.