Re: New egg: CHICKEN Transducers

[Jeremy Steward]

On 1/5/23 00:44, Peter Bex wrote:
> > And I've written a short blog post outlining some of my frustrations that
> > led me to writing this egg:
> >
> > <https://www.thatgeoguy.ca/blog/2023/01/04/reflections-on-transducers/>
>
> Excellent observations, especially regarding the fold argument inconsistency,
> that's so cringeworthy it isn't even funny.  I thoroughly enjoyed reading
> the post.  Even if it's critical of Scheme, it's done in a loving way,
> as you mention ;)  I also agree with you that SRFI has probably had its
> best time.

Thanks for the kind words, I do hope it is recognized that what I am 
saying in the post is with love for Scheme. The fold thing gets me every 
time in every language I ever work in, so it was somewhat cathartic to 
write that down.

> > Happy to engage with the rest of the community on what the next priority for
> > such a library should be. My hope is that this library helps us all move
> > away from XXX-map and XXX-filter procedures for each individual type.
>
> Well, like I said I kinda prefer having the type-specific procedures.
> I rarely deal with situations where I don't know the type on which I'm
> operating.  I usually treat generic code with suspicion, as it's not
> going to be as fast as it could be, and higher-order code tends to be
> more difficult to follow than necessary.

Well, the final transducers produced /are/ type-specific. The input and 
output types are parameterized by the folder and collector, 
respectively. This is a bit different than Clojure where protocols are 
used to paper over the types entirely. E.g.

    (transduce list-fold
               (map add1)
               (collect-vector)
               (list 1 2 3))

You will get a type error here from |list-fold| if you pass in a vector 
in place of a list in the last position, for example. Likewise, 
|collect-vector| can only ever collect a vector, and how fast it is 
depends on the size & number of re-allocations (hence why it collects in 
amortized O(1) time, and has an optional size-hint).

I tried my best to avoid any of the higher-level performance pitfalls 
that would come about with transducers. Of course, this doesn't mean bad 
code can't be subbed in, but this is a library with versions and an 
issue tracker and a repo, not just a spec. :) I fully welcome anybody to 
tell me how to make things more performant or easier to debug if there's 
good ideas lying around.

You are right though that higher-order code does tend towards being more 
difficult to follow. My hope is that as long as the parameterizations in 
transducers (as they currently exist) remain explicit that this can be 
mostly avoided.

> However, transducers offer one thing that the type-specific procedures
> do not - pipelining without accumulating lots of intermediate results.
> For me that's the killer feature of transducers.

Yes, the ability to avoid intermediates is perhaps the main advantage. I 
also think that the fact that collection is distinct from folding and 
transducing is a separate advantage -- e.g. if you know that you need a 
vector at the end there's no need to worry about whether that aligns 
with the input / processing steps.

Regards, and thanks again for the thoughtful reply,
--
Jeremy Steward