Re: [Discuss-gnuradio] [RFC] PMT succession: experts' opinions on serialization libs

[Nick Foster]

I think it's wise to avoid hitching our metaphorical horse to any software project we can't absolutely depend on; questionable dependency choices have bit us, as you mention, many times.

With that (somewhat) in mind, what about Flatbuffers as an alternative to Cap'n Proto? Seems much more actively maintained, more widely used, and designed with the same goals in mind. No parsing step required, so performance is good. Memory use seems to be similarly efficient.

Now, I haven't actually used Flatbuffers before, so anyone who has experience with them (or your other favorite serialization library!), please chime in here.

Nick

On Sat, Sep 28, 2019 at 2:02 PM Marcus Müller <address@hidden> wrote:

To avoid the setting in of post-Conference downs:

Dearest SDR Community with the Highest count of "Awesome" in the
Universe,

discussions at GRCon have made it abundantly clear that PMT needs to be
replaced¹.
Main issues with PMT are:

* broken type system,
* low suitability as tool for internal packeted data,
* lack of interoperability with anything but the languages we support
natively,
* platform-dependency of the serialization format,
* bugs.

(to a far lesser degree: performance.)

Having talked to multiple domain experts in networked systems and
heterogenous cooperative platforms, I've come to the conclusion,
furthermore, that in the interest of RPC'ing everything², the structure
of messages exchanged between blocks should be predefined.

In this sense, PMT's (limited) ability do describe itself is of no help
– instead, it introduces ambiguity.

The dominant choice for binary serialization with known schemata is
Proto Buffers; however, they require deserialization of a whole message
if there's substructures before the field you want to access³.

Cap'n Proto[1] (by one of the original authors of protobufs) seems to
be a suitable alternative, and comparing it to its competitors, it
seems to have achieved a significant level of maturity and has at least
one large-scale adopter⁴.

Among its advantages seems to be that operations on the message happen
in-place (i.e. there's actually a compatible C++ struct representation
of the memory layout), which rendered it my favourite library for this
job.

Now, problem: I've never personally worked with Cap'n Proto, or
protobuf, or any modern, non-self-describing serialization library;
I've got minimal experience with MessagePack, but as said, that's not
probably going to make it due to lack of schema.

So, here I am, calling for your experience! Hurl ins(igh|ul)ts at me,
preferable here on this list.

Best regards,
Marcus


[1] https://capnproto.org/index.html

¹This doesn't come as a surprise – it's just a good point in time to
point this out.
²You've probably noticed that we're working towards a less machine-
dependent, more portable, distributable computational model
³This can become problematic for messages with large data blobs in
between, or very complex protocol headers
⁴CloudFlare seems to be using Cap'n Proto for Web workers, for example.
However, it's but at revision 0.7, which indeed implies a bit of
headache – other tools that never reached 1.0 that we rely on include
Thrift, and that's really not a club that you'd want to be in.


_______________________________________________
Discuss-gnuradio mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio