discuss-gnuradio
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Discuss-gnuradio] [RFC] PMT succession: experts' opinions on serial


From: West, Nathan
Subject: Re: [Discuss-gnuradio] [RFC] PMT succession: experts' opinions on serialization libs
Date: Sun, 29 Sep 2019 09:32:59 -0500

While developing libsigmf (https://github.com/deepsig/libsigmf) I went through a similar experience. The objective was to have painless json file/string <-> real c++ type (this means the serialization thing had to generate a slimmed down type that is effectively a struct of vector, int, float, etc). I looked at cap'n proto, flatbuffers, protobuf, and some others that weren't as mature/supported. I actually started with cap'n proto and had to switch to flatbuffers.

Other bonuses I encountered were the whole "zero-time serialization" (In my experience people love making fun of cap'n proto for this, but it's true for all of the popular architectures and then you pay for shuffling bytes on more exotic architectures) and the sort of random-access parsing. A big perk of cap'n proto was it actually came with json-parsing to flatbuffer type *and* had the ability to generate a sane struct definition.

I eventually switched to flatbuffers. Here's a randomly-ordered list of why

* cap'n proto's author is no doubt an incredible programmer. However, he made some really weird choices. Like wrote his own string class in libkj and made all of cap'n proto depend on it. And it's not all that friendly with c++ string. So you wind up having to convert every single string in to a kj::string just for serialization (and vice versa)
* libkj also implements its own memory allocators, filesystem library (ala boost filesystem / c++ filesystem), mutexes, and threads. I'm not capable of vetting those implementation but beg to be asked a big old "WHY?!"
* cap'n proto is incredibly strict (in CS parlance I guess you can call it opinionated) on protocol description language. It enforces things such as FirstletterMustBeCapital
* I also felt the syntax of moving around these objects was a bit awkward

I think there may be some other less appalling things that I've struck from my memory as "writing your own string that's incompatible with the rest of c++" is a big enough grievance that I'm only just now starting to forgive gcc dual abi for (but at least even that is at the ABI level which is always hard to get right). Flatbuffers offered pretty much the same promises as cap'n proto but without the "reinvent the standard library first" mindset and let's you make your own choices about how to name your fields, etc. I will say cap'n proto had a nice-looking extension capability to their parsing which I haven't found a good alternative for.

The downside is flatbuffers is released by google/android's benevolence and somewhat competes with protobufs which is clearly their darling. I can't tell how much anyone else cares about flatbuffers although the *major* advantage to flatbuffers that I couldn't find in protobuf (which made me choose flatbuffers) is the struct code generator so you get out pretty normal struct's from their code generator that look like something you would write if you did it by hand.

The downside is I don't know what complex<T> support looks like in any of these things.

Cheers,
Nathan

On Sun, Sep 29, 2019 at 5:21 AM Marcus Müller <address@hidden> wrote:
Hi Nick,


On Sat, 2019-09-28 at 15:54 -0700, Nick Foster wrote:
> I think it's wise to avoid hitching our metaphorical horse to any
> software project we can't absolutely depend on; questionable
> dependency choices have bit us, as you mention, many times.

Sad but true.

> With that (somewhat) in mind, what about Flatbuffers as an
> alternative to Cap'n Proto? Seems much more actively maintained, more
> widely used, and designed with the same goals in mind. No parsing
> step required, so performance is good. Memory use seems to be
> similarly efficient.

See, that's the kind of input I was looking for!
So, what I don't like about flatbuffers was the lack of field IDs in
the schema; relying on new fields only being added to the end of a
structure.

However, thanks to your encouragement I've revisited that and:
Flatbuffers *does* do IDs, if you want (it's slightly awkward, but all
formal structure definition languages are, aside from ASN.1 ;) ).

Regarding efficiency: https://github.com/thekvs/cpp-serializers seems
to agree with your assessment.

What flatbuffers really got going for it compared to the Cap'n is that
docs are structured instead of "cool feature-oriented". What Cap'n
Proto has is the more sympathetic code base, but maybe I shouldn't be
applying too many human qualities to code bases. gnuradio-runtime/lib/
gets jealous everytime I do that.

Best regards,
Marcus

>
> Now, I haven't actually used Flatbuffers before, so anyone who has
> experience with them (or your other favorite serialization library!),
> please chime in here.
>
> Nick
>
> On Sat, Sep 28, 2019 at 2:02 PM Marcus Müller <address@hidden>
> wrote:
> > To avoid the setting in of post-Conference downs:
> >
> > Dearest SDR Community with the Highest count of "Awesome" in the
> > Universe,
> >
> > discussions at GRCon have made it abundantly clear that PMT needs
> > to be
> > replaced¹.
> > Main issues with PMT are:
> >
> > * broken type system,
> > * low suitability as tool for internal packeted data,
> > * lack of interoperability with anything but the languages we
> > support
> > natively,
> > * platform-dependency of the serialization format,
> > * bugs.
> >
> > (to a far lesser degree: performance.)
> >
> > Having talked to multiple domain experts in networked systems and
> > heterogenous cooperative platforms, I've come to the conclusion,
> > furthermore, that in the interest of RPC'ing everything², the
> > structure
> > of messages exchanged between blocks should be predefined.
> >
> > In this sense, PMT's (limited) ability do describe itself is of no
> > help
> > – instead, it introduces ambiguity.
> >
> > The dominant choice for binary serialization with known schemata is
> > Proto Buffers; however, they require deserialization of a whole
> > message
> > if there's substructures before the field you want to access³.
> >
> > Cap'n Proto[1] (by one of the original authors of protobufs) seems
> > to
> > be a suitable alternative, and comparing it to its competitors, it
> > seems to have achieved a significant level of maturity and has at
> > least
> > one large-scale adopter⁴.
> >
> > Among its advantages seems to be that operations on the message
> > happen
> > in-place (i.e. there's actually a compatible C++ struct
> > representation
> > of the memory layout), which rendered it my favourite library for
> > this
> > job.
> >
> > Now, problem: I've never personally worked with Cap'n Proto, or
> > protobuf, or any modern, non-self-describing serialization library;
> > I've got minimal experience with MessagePack, but as said, that's
> > not
> > probably going to make it due to lack of schema.
> >
> > So, here I am, calling for your experience! Hurl ins(igh|ul)ts at
> > me,
> > preferable here on this list.
> >
> > Best regards,
> > Marcus
> >
> >
> > [1] https://capnproto.org/index.html
> >
> > ¹This doesn't come as a surprise – it's just a good point in time
> > to
> > point this out.
> > ²You've probably noticed that we're working towards a less machine-
> > dependent, more portable, distributable computational model
> > ³This can become problematic for messages with large data blobs in
> > between, or very complex protocol headers
> > ⁴CloudFlare seems to be using Cap'n Proto for Web workers, for
> > example.
> > However, it's but at revision 0.7, which indeed implies a bit of
> > headache – other tools that never reached 1.0 that we rely on
> > include
> > Thrift, and that's really not a club that you'd want to be in.
> >
> >
> > _______________________________________________
> > Discuss-gnuradio mailing list
> > address@hidden
> > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio


_______________________________________________
Discuss-gnuradio mailing list
address@hidden
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

reply via email to

[Prev in Thread] Current Thread [Next in Thread]