Hi all.
I was contemplating how to design a new file format for schemes that has more bang's and whistles that the byte-stream and at the same time lend itself well to some kind of streaming between smaller utility programs piped together.
The obvious solution is to use (read) together whith static scheme data, but I was thinking more features. I would like to have variables and I would like the streams to be able to do delayed evaluations and have backtracking information.
Variables:
Sometimes it would be nice to add modifications of the data at the end of the stream. For example you hand out a document to 10 persons and they all edit the document and return their versions of it. If the modifications ar just added at the end of the document
as a serious of variable-set! we can compare the old and all the new versions quite easilly in a tool. Variables have also other nice features like the ability to add the binding later in a stream algorithm.
Backtracking:
This can be handy to add undo points to the stream and persist this information
Delayed Evaluation:
This is an interesting feature. Assume a sequence of indata x1,x2,x3. and that the value of x1 is decided at the evaluation of x3 in
order for the stream to not be out of sync one would like to be able to output a variable at the x1 position and add a hook that calulate the value when x3 get's it's value. This problem can be solved elegantly and quite generally by using variables and closures. Therefore we would appart from the usual static data add closures with the closed over variables beeing in the same class and where any variable reference is refering to a variable defined earlier in the sequence.
The cool thing is that all this (almost) already has an efficient implementation in gule-log together with a large set of utility funcitons. I would even expect that reading in such a stream into guile will be faster then reading in scheme due to the fact that everything is allocated in a stack and no scm_cons need to be used. What's neeeded to be done is some code for writing the stack back and forth to disk and add some utility code to do streaming. After that it can be playable.
WDYT