|
| From: | West, Nathan |
| Subject: | Re: [Discuss-gnuradio] [RFC] PMT succession: experts' opinions on serialization libs |
| Date: | Sun, 29 Sep 2019 16:53:55 -0500 |
Wow, thanks. Yeah, that clearly reinforces flatbuffers as tool of
choice.
Tbh, I'd just assume that in all these formats, being tight-packing by
default, std::complex<float> can just be represented by the equivalent
of struct {float re; float im;} complex;.
On Sun, 2019-09-29 at 09:32 -0500, West, Nathan wrote:
> While developing libsigmf (https://github.com/deepsig/libsigmf) I
> went through a similar experience. The objective was to have painless
> json file/string <-> real c++ type (this means the serialization
> thing had to generate a slimmed down type that is effectively a
> struct of vector, int, float, etc). I looked at cap'n proto,
> flatbuffers, protobuf, and some others that weren't as
> mature/supported. I actually started with cap'n proto and had to
> switch to flatbuffers.
>
> Other bonuses I encountered were the whole "zero-time serialization"
> (In my experience people love making fun of cap'n proto for this, but
> it's true for all of the popular architectures and then you pay for
> shuffling bytes on more exotic architectures) and the sort of random-
> access parsing. A big perk of cap'n proto was it actually came with
> json-parsing to flatbuffer type *and* had the ability to generate a
> sane struct definition.
>
> I eventually switched to flatbuffers. Here's a randomly-ordered list
> of why
>
> * cap'n proto's author is no doubt an incredible programmer. However,
> he made some really weird choices. Like wrote his own string class in
> libkj and made all of cap'n proto depend on it. And it's not all that
> friendly with c++ string. So you wind up having to convert every
> single string in to a kj::string just for serialization (and vice
> versa)
> * libkj also implements its own memory allocators, filesystem library
> (ala boost filesystem / c++ filesystem), mutexes, and threads. I'm
> not capable of vetting those implementation but beg to be asked a big
> old "WHY?!"
> * cap'n proto is incredibly strict (in CS parlance I guess you can
> call it opinionated) on protocol description language. It enforces
> things such as FirstletterMustBeCapital
> * I also felt the syntax of moving around these objects was a bit
> awkward
>
> I think there may be some other less appalling things that I've
> struck from my memory as "writing your own string that's incompatible
> with the rest of c++" is a big enough grievance that I'm only just
> now starting to forgive gcc dual abi for (but at least even that is
> at the ABI level which is always hard to get right). Flatbuffers
> offered pretty much the same promises as cap'n proto but without the
> "reinvent the standard library first" mindset and let's you make your
> own choices about how to name your fields, etc. I will say cap'n
> proto had a nice-looking extension capability to their parsing which
> I haven't found a good alternative for.
>
> The downside is flatbuffers is released by google/android's
> benevolence and somewhat competes with protobufs which is clearly
> their darling. I can't tell how much anyone else cares about
> flatbuffers although the *major* advantage to flatbuffers that I
> couldn't find in protobuf (which made me choose flatbuffers) is the
> struct code generator so you get out pretty normal struct's from
> their code generator that look like something you would write if you
> did it by hand.
>
> The downside is I don't know what complex<T> support looks like in
> any of these things.
>
> Cheers,
> Nathan
>
> On Sun, Sep 29, 2019 at 5:21 AM Marcus Müller <address@hidden>
> wrote:
> > Hi Nick,
> >
> >
> > On Sat, 2019-09-28 at 15:54 -0700, Nick Foster wrote:
> > > I think it's wise to avoid hitching our metaphorical horse to any
> > > software project we can't absolutely depend on; questionable
> > > dependency choices have bit us, as you mention, many times.
> >
> > Sad but true.
> >
> > > With that (somewhat) in mind, what about Flatbuffers as an
> > > alternative to Cap'n Proto? Seems much more actively maintained,
> > more
> > > widely used, and designed with the same goals in mind. No parsing
> > > step required, so performance is good. Memory use seems to be
> > > similarly efficient.
> >
> > See, that's the kind of input I was looking for!
> > So, what I don't like about flatbuffers was the lack of field IDs
> > in
> > the schema; relying on new fields only being added to the end of a
> > structure.
> >
> > However, thanks to your encouragement I've revisited that and:
> > Flatbuffers *does* do IDs, if you want (it's slightly awkward, but
> > all
> > formal structure definition languages are, aside from ASN.1 ;) ).
> >
> > Regarding efficiency: https://github.com/thekvs/cpp-serializers
> > seems
> > to agree with your assessment.
> >
> > What flatbuffers really got going for it compared to the Cap'n is
> > that
> > docs are structured instead of "cool feature-oriented". What Cap'n
> > Proto has is the more sympathetic code base, but maybe I shouldn't
> > be
> > applying too many human qualities to code bases. gnuradio-
> > runtime/lib/
> > gets jealous everytime I do that.
> >
> > Best regards,
> > Marcus
> >
> > >
> > > Now, I haven't actually used Flatbuffers before, so anyone who
> > has
> > > experience with them (or your other favorite serialization
> > library!),
> > > please chime in here.
> > >
> > > Nick
> > >
> > > On Sat, Sep 28, 2019 at 2:02 PM Marcus Müller <
> > address@hidden>
> > > wrote:
> > > > To avoid the setting in of post-Conference downs:
> > > >
> > > > Dearest SDR Community with the Highest count of "Awesome" in
> > the
> > > > Universe,
> > > >
> > > > discussions at GRCon have made it abundantly clear that PMT
> > needs
> > > > to be
> > > > replaced¹.
> > > > Main issues with PMT are:
> > > >
> > > > * broken type system,
> > > > * low suitability as tool for internal packeted data,
> > > > * lack of interoperability with anything but the languages we
> > > > support
> > > > natively,
> > > > * platform-dependency of the serialization format,
> > > > * bugs.
> > > >
> > > > (to a far lesser degree: performance.)
> > > >
> > > > Having talked to multiple domain experts in networked systems
> > and
> > > > heterogenous cooperative platforms, I've come to the
> > conclusion,
> > > > furthermore, that in the interest of RPC'ing everything², the
> > > > structure
> > > > of messages exchanged between blocks should be predefined.
> > > >
> > > > In this sense, PMT's (limited) ability do describe itself is of
> > no
> > > > help
> > > > – instead, it introduces ambiguity.
> > > >
> > > > The dominant choice for binary serialization with known
> > schemata is
> > > > Proto Buffers; however, they require deserialization of a whole
> > > > message
> > > > if there's substructures before the field you want to access³.
> > > >
> > > > Cap'n Proto[1] (by one of the original authors of protobufs)
> > seems
> > > > to
> > > > be a suitable alternative, and comparing it to its competitors,
> > it
> > > > seems to have achieved a significant level of maturity and has
> > at
> > > > least
> > > > one large-scale adopter⁴.
> > > >
> > > > Among its advantages seems to be that operations on the message
> > > > happen
> > > > in-place (i.e. there's actually a compatible C++ struct
> > > > representation
> > > > of the memory layout), which rendered it my favourite library
> > for
> > > > this
> > > > job.
> > > >
> > > > Now, problem: I've never personally worked with Cap'n Proto, or
> > > > protobuf, or any modern, non-self-describing serialization
> > library;
> > > > I've got minimal experience with MessagePack, but as said,
> > that's
> > > > not
> > > > probably going to make it due to lack of schema.
> > > >
> > > > So, here I am, calling for your experience! Hurl ins(igh|ul)ts
> > at
> > > > me,
> > > > preferable here on this list.
> > > >
> > > > Best regards,
> > > > Marcus
> > > >
> > > >
> > > > [1] https://capnproto.org/index.html
> > > >
> > > > ¹This doesn't come as a surprise – it's just a good point in
> > time
> > > > to
> > > > point this out.
> > > > ²You've probably noticed that we're working towards a less
> > machine-
> > > > dependent, more portable, distributable computational model
> > > > ³This can become problematic for messages with large data blobs
> > in
> > > > between, or very complex protocol headers
> > > > ⁴CloudFlare seems to be using Cap'n Proto for Web workers, for
> > > > example.
> > > > However, it's but at revision 0.7, which indeed implies a bit
> > of
> > > > headache – other tools that never reached 1.0 that we rely on
> > > > include
> > > > Thrift, and that's really not a club that you'd want to be in.
> > > >
> > > >
> > > > _______________________________________________
> > > > Discuss-gnuradio mailing list
> > > > address@hidden
> > > > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
> >
> >
> > _______________________________________________
> > Discuss-gnuradio mailing list
> > address@hidden
> > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>
> _______________________________________________
> Discuss-gnuradio mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
| [Prev in Thread] | Current Thread | [Next in Thread] |