Re: Announcement stis-engine

[Stefan Israelsson Tampe]

HI all,

I added a new gem, it' a c based serializer and deserializer of guile data which is relative feature rich and will ping pong 

between scheme and c-land in order to be able to be fiber friendly. So currently the benchmarks are for the following copy

operations,

,time (load (dump (make-bytevector N 1))

1s for ~ 1GB (test of memcpy)

,time (load (dump (make-vector N 1))

1s for ~ 100MB (will be less pipelined)

,time (load (dump (iota N))

1s for ~ 50MB (will create in total N cons cells with 5 bytes)

The scheme based version of these serializers and deserializers was able to perform 5MB per second, but thats with much higher allocation overhead

This means that we match pythons dump and load functions. I will add those to my fibers piping tool

Currently supported data structures

all numbers yes even bignums

all immediates  ('(), booleans characters and whatnot)

bytevectors

strings (wide strings and narrow strings)

symbols

keywords

pairs

vectors

the location of the c-code is in 

guile-persist:   https://gitlab.com/tampe/guile-persist

(use-modules (persist persistance)) => atom-load-bv and atom-dump-bv is defined.

This library is used by python-on-guile and I will replace the current dump and load with this in the end. Also this tool

will be added to the fiber piping system I am working with in stis-engine.

On Sat, Aug 14, 2021 at 2:07 PM Stefan Israelsson Tampe <stefan.itampe@gmail.com> wrote:

Next is to add support for

1. self referential data structures like #1 = '(1 2 3 #1), string compression

2. Structs

3 .Goops Classes and Objects

Then I will add

1. Lambdas

and then Continuations to be able to serialize (if you turn off unboxing optimizationsas in guile)

After that a toolbox to customize tserializing just as with python

Note all this is supported in python-on-guile, just that the new code is better suited for proper streaming 

and the code is much nicer in my personal view so you do have all these features already if you use pythons

dumps and loads commands together with string->fpipe and fpipe->string.  

The new thing is essentially stream compression and streams to zmq's. Stream compression wll come to 

python as well as the zlib python module is very rudementary