Re: Let BIO also handle already allocated buffers

[Christian Grothoff]

Hi Alessio,

I'm a bit torn about your proposal. It makes sense for BIO to support
to-memory serialization instead of just to-file.

However, there are two issues. First, your patch introduces a new type
'struct GNUNET_BIO_MemoryHandle'. I think to avoid having to duplicate
all of the individual BIO-read/write functions, the 'in memory' vs 'in
file' should just be a different function to create the same 'struct
GNUNET_BIO_ReadHandle' or 'GNUNET_BIO_WriteHande' we already have, just
this time backed by just buffer instead of a file. [This is what for me
makes the patch not really ready to merge at an API-level.]

Second, as I've mentioned to some people before but certainly not to
you, I've considerd the BIO API to be an abomination (for at least 5
years now) that we should likely re-do completely, and so I'm VERY
hesitant to see more code use that horrible style ;-).

Instead of (to take an example from hostlist):

        ((GNUNET_OK == GNUNET_BIO_read_string (rh, "url", &uri,
MAX_URL_LEN)) &&
         (NULL != uri) &&
         (GNUNET_OK == GNUNET_BIO_read_int32 (rh, &times_used)) &&
         (GNUNET_OK == GNUNET_BIO_read_int64 (rh, &quality)) &&
         (GNUNET_OK == GNUNET_BIO_read_int64 (rh, &last_used)) &&
         (GNUNET_OK == GNUNET_BIO_read_int64 (rh, &created)) &&
         (GNUNET_OK == GNUNET_BIO_read_int32 (rh, &hellos_returned)))

I would prefer to see something like:

struct GNUNET_BIO_ReadSpec spec[] = {
  GNUNET_BIO_read_string (&uri, MAX_URL_LEN),
  GNUNET_BIO_read_uint32 (&times_used),
  GNUNET_BIO_read_uint64 (&quality),
  GNUNET_BIO_read_abs_time (&last_used)
  GNUNET_BIO_read_end()
};

if (GNUNET_OK !=
    GNUNET_BIO_run (bio, spec))
  ...

more similar to the gnunet{pq,sq,my,json} APIs we have that use similar
'spec' arrays to serialize from/to database and or JSON.

bio = GNUNET_BIO_open_file(fd); // and
bio = GNUNET_BIO_open_buffer(buf, buf_size);

would then both of course make sense to have.


Note that it the above construction could likely even be made to support
arrays in a reasonably sane way:
  GNUNET_BIO_write_array (array_len,
                          array_base,
                          sizeof (*array_base),
    GNUNET_BIO_write_uint32 (NULL+offset),
    GNUNET_BIO_write_uint64 (NULL+offset),
    GNUNET_BIO_read_end())

where the 'write_array' would first write the
'array_len', and then iterate over the 'inner'
specification (given as varargs!), adding to
each 'offset' the array_base + i*sizeof(*array_base).
With a bit of macros the syntax could even be made pretty nice ...

OTOH, I'm also wondering if we're still not re-inventing ASN.1 badly here...

My 2 cents

Happy hacking!

Christian


On 4/18/20 4:34 PM, Alessio Vanni wrote:
> Hello,
> 
> aside from working on files by buffering I/O, the BIO library also does
> *formatted* I/O, like serializing strings or integers.
> 
> I found myself in a situation where my client and my service are
> communicating using variable-length messages (containing a sequence of
> variable-length strings, mostly), so I'd have to parse the messages "by
> hand" using some kind of serialization.  Since BIO has this formatted
> I/O, I thought I could "offload" this de/serialization to the library,
> but it works only on files and writing the message to a file and reading
> it back again is suboptimal.
> 
> As such, I hacked together a set of functions in BIO to work with
> already allocated buffers, such as when dealing with messages received
> from a message queue.  The patch is attached with this mail.  Right now
> the API is written to cover my needs, so it might be lacking in some
> parts.
> 
> Thanks,
> A.V.
>

signature.asc
Description: OpenPGP digital signature