Re: [libmicrohttpd] Problem with HTTP/1.1 chunked packets when connection is about to close

[Iron Bug at gmail.com]

Greetings again.

But I take it, the chunked transfer is the only way for sending large files 
that cannot/should not fit in buffer in RAM. Of course, I set unspecified size 
and intentionally use chunked transfer. Moreover, I use pipelining for 
transferring many chunked files in one TCP session and close it when client 
doesn't need it anymore to avoid dangling TCP sessions that system may keep 
open for quite long time which is not desireable in my case. I optimize my 
server part for memory print and it should not read the whole files content 
into RAM. So chunked encoding is all right and very useful.

The proper behavior (I specially checked in standard) when a client sends a 
request with "Connection: close" header, the server should complete the 
response (whether it's chunked or not) to the request and then close the 
connection. And everything works fine until the "Connection: close" request 
comes: then server misses the "Transfer-Encoding: chunked" header for the last 
response. As I take it, it only has to check that connection is not closed yet, 
it should be closed at the end of transfer, after processing the last client's 
request. So it's not a bug when application forcibly sets chunked encoding when 
connection is closed, it's just a hack to fix the real bug with lost header in 
chunked transfer that should be ended with closing connection.

Btw, if I write here, what do you think about HTTP/2? I found 
libmicrohttpd-http2 on the net, but it was a bit outdated and a little buggy. I 
fixed bugs and merged both http2 and the latest libmicrohttpd versions 
together, so now I use the extended version that supports both HTTP/1.x and 
HTTP/2. But I wonder if HTTP/2 (or HTTP/3 that is a draft yet but is already 
implemented in main browsers) are going to be supported in the library?

Sincerely, Yana A. Kireyonok aka Iron Bug

On Mon, 26 Apr 2021 12:00:39 -0400
libmicrohttpd-request@gnu.org wrote:

> Send libmicrohttpd mailing list submissions to
>       libmicrohttpd@gnu.org
> 
> To subscribe or unsubscribe via the World Wide Web, visit
>       https://lists.gnu.org/mailman/listinfo/libmicrohttpd
> or, via email, send a message with subject or body 'help' to
>       libmicrohttpd-request@gnu.org
> 
> You can reach the person managing the list at
>       libmicrohttpd-owner@gnu.org
> 
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of libmicrohttpd digest..."
> 
> 
> Today's Topics:
> 
>    1. Problem with HTTP/1.1 chunked packets when connection is
>       about to close (Iron Bug at gmail.com)
>    2. Re: Problem with HTTP/1.1 chunked packets when connection is
>       about to close (Evgeny Grin)
> 
> 
> ----------------------------------------------------------------------
> 
> Message: 1
> Date: Sun, 25 Apr 2021 21:45:58 +0500
> From: "Iron Bug at gmail.com" <death.iron.bug@gmail.com>
> To: libmicrohttpd@gnu.org
> Subject: [libmicrohttpd] Problem with HTTP/1.1 chunked packets when
>       connection is about to close
> Message-ID: <20210425214558.c6cd4b410fc806e70505d80e@gmail.com>
> Content-Type: text/plain; charset=US-ASCII
> 
> Greetings.
> 
> I use libmicrohttpd in my project and I found one bug in a very specific 
> situation with chunked messages.
> 
> The scheme I use is keep-alive connection for transferring some files 
> (chunked content, HTTP/1.1 pipelining). When transfer is over, connection is 
> closed by client request.
> 
> I create responses with MHD_create_response_from_callback and the HTTP header 
> "Transfer-Encoding: chunked" is automatically added to packet headers. But 
> when the transfer is over and the last file requested, client sends 
> "Connection: close". Theoretically, by HTTP standard, the server should 
> process the request and then close the socket. And it does so, it sends the 
> chunked data and closes the connection. But in the response to the last 
> request it loses the "Transfer-Encoding: chunked" header and the client fails 
> to accept the wrong formatted HTTP packet. If I manually add 
> "Transfer-Encoding: chunked" to the last HTTP response, it works fine. 
> 
> I think the problem hides in this code piece:
> 
>       file src/microhttpd/connection.c, line 1376, in function 
> have_chunked_upload:
> -----------------------------------------------------------------
>       if ( (MHD_NO != keepalive_possible (connection)) &&
>            (MHD_str_equal_caseless_ (MHD_HTTP_VERSION_1_1,
>                                      connection->version) ) )
>       {
>         if (NULL == have_encoding)
>         {
>           must_add_chunked_encoding = true;
>           connection->have_chunked_upload = true;
>         }
>       ...
> -----------------------------------------------------------------
> 
> The connection is marked as non keep-alive at the moment after "Connection: 
> close" and the content is still chunked. But the flag is not set and then the 
> error occurs.
> 
> Sincerely, Yana A. Kireyonok aka Iron Bug
> 
> 
> 
> ------------------------------
> 
> Message: 2
> Date: Mon, 26 Apr 2021 16:42:20 +0300
> From: Evgeny Grin <k2k@yandex.ru>
> To: libmicrohttpd@gnu.org
> Subject: Re: [libmicrohttpd] Problem with HTTP/1.1 chunked packets
>       when connection is about to close
> Message-ID: <67616383-2a38-5ddc-353c-696456da1629@yandex.ru>
> Content-Type: text/plain; charset="utf-8"; Format="flowed"
> 
> Привет, Яна!
> 
> Actually this behaviour is not a bug.
> Chunked encoding produces binary larger output and makes heavier system 
> and network load, so MHD tries to avoid chunked output if possible.
> 
> If you provide response size as the first argument for 
> MHD_create_response_from_callback() MHD will never use chunked encoding.
> 
> If you provide "-1" (MHD_SIZE_UNKNOWN) as the size then MHD will follow 
> the next logic:
> * if connection is "keep-alive" then only choice is chunked encoding and 
> chunked encoding is used,
> * if connection have to be closed after sending response then end of 
> response can be indicated by end of connection so "heavier" chunked 
> encoding is not used.
> 
> This logic most probably will be re-considered soon as latest HTTP/1.1 
> specification forbids responses without both encoding and content-size 
> (and ending connection without specifying content size is unreliable).
> 
> However I discovered bug related to chunked encoding (when application 
> forcibly sets "Transfer-Encoding: chunked") and "Connection: close". It 
> will be fixed in the next MHD release.
> 
> -- 
> Wishes,
> Evgeny
> 
> On 25.04.2021 19:45, Iron Bug at gmail.com wrote:
> > Greetings.
> > 
> > I use libmicrohttpd in my project and I found one bug in a very specific 
> > situation with chunked messages.
> > 
> > The scheme I use is keep-alive connection for transferring some files 
> > (chunked content, HTTP/1.1 pipelining). When transfer is over, connection 
> > is closed by client request.
> > 
> > I create responses with MHD_create_response_from_callback and the HTTP 
> > header "Transfer-Encoding: chunked" is automatically added to packet 
> > headers. But when the transfer is over and the last file requested, client 
> > sends "Connection: close". Theoretically, by HTTP standard, the server 
> > should process the request and then close the socket. And it does so, it 
> > sends the chunked data and closes the connection. But in the response to 
> > the last request it loses the "Transfer-Encoding: chunked" header and the 
> > client fails to accept the wrong formatted HTTP packet. If I manually add 
> > "Transfer-Encoding: chunked" to the last HTTP response, it works fine.
> > 
> > I think the problem hides in this code piece:
> > 
> >        file src/microhttpd/connection.c, line 1376, in function 
> > have_chunked_upload:
> > -----------------------------------------------------------------
> >        if ( (MHD_NO != keepalive_possible (connection)) &&
> >             (MHD_str_equal_caseless_ (MHD_HTTP_VERSION_1_1,
> >                                       connection->version) ) )
> >        {
> >          if (NULL == have_encoding)
> >          {
> >            must_add_chunked_encoding = true;
> >            connection->have_chunked_upload = true;
> >          }
> >     ...
> > -----------------------------------------------------------------
> > 
> > The connection is marked as non keep-alive at the moment after "Connection: 
> > close" and the content is still chunked. But the flag is not set and 
> then the error occurs.
> > 
> > Sincerely, Yana A. Kireyonok aka Iron Bug
> > 
> 
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: OpenPGP_signature
> Type: application/pgp-signature
> Size: 840 bytes
> Desc: OpenPGP digital signature
> URL: 
> <https://lists.gnu.org/archive/html/libmicrohttpd/attachments/20210426/955bfc0a/attachment.sig>
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> _______________________________________________
> libmicrohttpd mailing list
> libmicrohttpd@gnu.org
> https://lists.gnu.org/mailman/listinfo/libmicrohttpd
> 
> 
> ------------------------------
> 
> End of libmicrohttpd Digest, Vol 144, Issue 2
> *********************************************


-- 
Iron Bug at gmail.com <death.iron.bug@gmail.com>