Re: [Discuss-gnuradio] [RFC] PMT succession: experts' opinions on serialization libs

[Marcus Müller]

Hi Nick,


On Sat, 2019-09-28 at 15:54 -0700, Nick Foster wrote:
> I think it's wise to avoid hitching our metaphorical horse to any
> software project we can't absolutely depend on; questionable
> dependency choices have bit us, as you mention, many times.

Sad but true.

> With that (somewhat) in mind, what about Flatbuffers as an
> alternative to Cap'n Proto? Seems much more actively maintained, more
> widely used, and designed with the same goals in mind. No parsing
> step required, so performance is good. Memory use seems to be
> similarly efficient.

See, that's the kind of input I was looking for!
So, what I don't like about flatbuffers was the lack of field IDs in
the schema; relying on new fields only being added to the end of a
structure.

However, thanks to your encouragement I've revisited that and:
Flatbuffers *does* do IDs, if you want (it's slightly awkward, but all
formal structure definition languages are, aside from ASN.1 ;) ).

Regarding efficiency: https://github.com/thekvs/cpp-serializers seems
to agree with your assessment.

What flatbuffers really got going for it compared to the Cap'n is that
docs are structured instead of "cool feature-oriented". What Cap'n
Proto has is the more sympathetic code base, but maybe I shouldn't be
applying too many human qualities to code bases. gnuradio-runtime/lib/
gets jealous everytime I do that.

Best regards,
Marcus

> 
> Now, I haven't actually used Flatbuffers before, so anyone who has
> experience with them (or your other favorite serialization library!),
> please chime in here.
> 
> Nick
> 
> On Sat, Sep 28, 2019 at 2:02 PM Marcus Müller <address@hidden>
> wrote:
> > To avoid the setting in of post-Conference downs:
> > 
> > Dearest SDR Community with the Highest count of "Awesome" in the
> > Universe,
> > 
> > discussions at GRCon have made it abundantly clear that PMT needs
> > to be
> > replaced¹.
> > Main issues with PMT are:
> > 
> > * broken type system,
> > * low suitability as tool for internal packeted data,
> > * lack of interoperability with anything but the languages we
> > support
> > natively,
> > * platform-dependency of the serialization format,
> > * bugs.
> > 
> > (to a far lesser degree: performance.)
> > 
> > Having talked to multiple domain experts in networked systems and
> > heterogenous cooperative platforms, I've come to the conclusion,
> > furthermore, that in the interest of RPC'ing everything², the
> > structure
> > of messages exchanged between blocks should be predefined.
> > 
> > In this sense, PMT's (limited) ability do describe itself is of no
> > help
> > – instead, it introduces ambiguity.
> > 
> > The dominant choice for binary serialization with known schemata is
> > Proto Buffers; however, they require deserialization of a whole
> > message
> > if there's substructures before the field you want to access³.
> > 
> > Cap'n Proto[1] (by one of the original authors of protobufs) seems
> > to
> > be a suitable alternative, and comparing it to its competitors, it
> > seems to have achieved a significant level of maturity and has at
> > least
> > one large-scale adopter⁴.
> > 
> > Among its advantages seems to be that operations on the message
> > happen
> > in-place (i.e. there's actually a compatible C++ struct
> > representation
> > of the memory layout), which rendered it my favourite library for
> > this
> > job.
> > 
> > Now, problem: I've never personally worked with Cap'n Proto, or
> > protobuf, or any modern, non-self-describing serialization library;
> > I've got minimal experience with MessagePack, but as said, that's
> > not
> > probably going to make it due to lack of schema.
> > 
> > So, here I am, calling for your experience! Hurl ins(igh|ul)ts at
> > me,
> > preferable here on this list.
> > 
> > Best regards,
> > Marcus
> > 
> > 
> > [1] https://capnproto.org/index.html
> > 
> > ¹This doesn't come as a surprise – it's just a good point in time
> > to
> > point this out.
> > ²You've probably noticed that we're working towards a less machine-
> > dependent, more portable, distributable computational model
> > ³This can become problematic for messages with large data blobs in
> > between, or very complex protocol headers
> > ⁴CloudFlare seems to be using Cap'n Proto for Web workers, for
> > example.
> > However, it's but at revision 0.7, which indeed implies a bit of
> > headache – other tools that never reached 1.0 that we rely on
> > include
> > Thrift, and that's really not a club that you'd want to be in.
> > 
> > 
> > _______________________________________________
> > Discuss-gnuradio mailing list
> > address@hidden
> > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio