Re: Chunked Encoding

[Ian Price]

ccing Daniel Hartwig, since he has been a bigger cheerleader for chunked
encoding than I have so far :).

Andy Wingo <address@hidden> writes:

> On_Thu 29 Sep 2011 16:09, Ian Price <address@hidden> writes:
>
>> If you've used the (web ...) modules, you may have noticed that guile
>> does not currently support chunked-encoding.
Sheesh that was a long time ago

>> What I did was introduce two new exported procedures for reading (all I
>> needed at the moment), namely 'read-chunk' and 'read-chunked-response-body'.
>>
>>   (read-chunk port)
>>   reads one chunk from a port.
>>
>>   (read-chunked-response-body response)
>>   read the full body for the response and returns it as a bytevector. It
>>   was written to be similar to 'read-response-body'.
>
> Is it possible to use soft ports?.  That would be nice, and it would
> allow client code to be able to read from the port until it gets EOF,
> without being concerned about the transfer-encoding.  Same thing goes
> for gzip/deflate/compress encoding.
It is possible, and I have some code for this, but I had been
procrastinating for a while because of a soft ports bug with flush, and
once that got fixed, just plain procrastinating.

>> Another option I've been thinking over would be to go for a sort of
>> chunking version of R6RS' 'transcoded-port' which would handle it
>> transparently for users of the returned port.
>
> Is this equivalent?  I don't have much experience with these.  It seems
> that custom binary input/output ports are more appropriate.
Well, what I meant is a port that would be layered over the top of
another. Soft ports or custom binary ports would be used to implement
it. (Is there a reason (effiencywise) to prefer one over the other?)


Basically, my interface right now is

(make-chunked-input-port port) -> input-port
(make-chunked-output-port port) -> output-port

These operate pretty much as you'd expect. The port returned from
'make-chunked-input-port' reads whole chunks from its argument port, and
maintains a buffer, from which it can satisfy smaller reads.
The port returned from 'make-chunked-output-port' buffers up the writes
and writes a whole (properly formatted) chunk on 'force-output'.

The only behaviour I'm not entirely sure of is what happens on
close. Let's be more concrete:

(define a <port>)
(define b (make-chunked-input-port a))
(close-port b)

What is the state of a?

(define c <port>)
(define d (make-chunked-output-port c))
(close-port d)

Likewise for c.


I think common practice in things like Java's BufferedReader would be to
have a be closed when b is. This may be undesirable since we could wish
to continue using the socket. On the other hand, once you start layering
ports, it is convenient to have the higher layers close the lower
layers. I think it might make sense to have a keyword argument,
#:dont-close? (or something), that specifies this behaviour, defaulting
to close.

c seems less clear to me. Again, once you have multiple layers, it would
be convenient, and it would properly handle any inner state saved up.

Any thoughts on this?

>
>> I'd also suggest extending 'http-get' from (web client) to handle
>> chunked encoding (and trailers too, I guess) for the user.
>
<snip>
>
> So, in the default case in which we do not mention "trailers" in the
> request, trailer fields should be strictly not required.  So the naive
> case should just work.
I think we can leave trailers until I have some actual data on how much
these are actually used in practice, and/or someone complains about it
being missing. WDYT?

FWIW I didn't see any code for handling them in Ruby's net/http.rb


>> Comments kindly requested,
>
> Again, very sorry for the delay!  I hope to be more on top of things
> this month.
I'm sorry for the delay too. Let's see if we can't finish it sometime in
the next week and end this once and for all :)

-- 
Ian Price

"Programming is like pinball. The reward for doing it well is
the opportunity to do it again" - from "The Wizardy Compiled"