Re: GWL pipelined process composition ?

[zimoun]

Hi Pjotr and Roel,

Thank you for the explanations.
I am not sure to have the skills to understand all of them.


> Yes, but the original question was whether you could stream data
> without writing to disk, right? Unix pipes are the system way of
> providing that functionality - with the added advantage of parallel
> processing between Unix processes. The downside, as you say is that it
> is not that com posable.

My original question was: pipelined process composition :-)
It is still my question ;-)

The unix pipes allow to compose at the procedure level, not at the
process level.


> Unsurprisingly there are no systems that handle that well - that I am
> aware. The best you'll get today is composable containers that 'talk'
> with each other. But that is ad hoc network programming.

Hum? I should miss a point.
I quickly look some stackoverflow questions about C implementation of
the shell pipe.
Perhaps, I need to test and fail by myself to understand why it is
hard to sequentially catch two stdout and then use them as stdin.
I mean, the single stdin write-on-disk version:
 y = f(x)  # process x and then write y
 z = g(y) # read y then process then write z
and the pipelined is: z = g(f(x)) where write 'y' is avoided.
This composition is done by the shell pipe.

The extension of two inputs is:
 r = f(p)
 s = h(q)
 z = g(r, s)
and the pipelined version: z = g(f(p), h(q)).

Why the fork+dup2 does not work to implement this "double" composition ?


> Adding composable pipes will magnify the size of the code base and
> make it fragile at the same time. Besides, network transport layers
> will add another possibility of IO bottle necks. It is a whole project
> in itself ;)

I trust you :-)

Even if I do not see why some network layers should be used to work
around the filesystem.
Maybe we are not talking about the same issue.
One is compose pipes in distributed memory context.
Another is compose pipes in shared memory context.

I am talking about compose pipes using one machine. :-)


> In FP you are working in one space, so it is much easier to compose
> (functions).

At the level of the processes, you are in one same space. It is the
space of files, i.e., the filesystem.


Now, after your explanations, the questions to myself are:
 1. is it possible to design (and implement with guile) shell pipes to
deal with two (or more) inputs ?
 2. is it possible to temporary mount something in the RAM to fake the
filesystem ?
 3. the already invented wheel is it not the Virtual FileSystem ?
Let me know your experienced insights :-)



All the best,
simon