[libmicrohttpd] 0.9.68 released

[Christian Grothoff]

Dear all,

I've now released libmicrohttpd 0.9.68, with the two wget-related
changes discussed on the list (allow insane clients that insist to
override content-length header, and fix empty-response HTTPS replies).
Also, I've fixed the bit-test logic in MHD_get_connection_values_n(),
expanded the test suite and won't generate the error message on
setsockopt() for UNIX domain sockets as suggested by Markus.

Happy hacking!

Christian

On 10/26/19 7:23 PM, Tim Rühsen wrote:
> Hi Christian,
> 
> that's wow and great ! Thank you so much :-)
> 
> I just pushed a change to SKIP the two affected tests with 0x00096701
> and 0x00096702.
> 
> But it would be nice to have release, else we'll likely see 0.9.67 going
> into several distributions and it would take a while for them to be
> updated then. E.g. Debian is still a 0.9.66 - and it would be great if
> they directly jump onto 0.9.68.
> 
> Regards, Tim
> 
> On 26.10.19 18:55, Christian Grothoff wrote:
>> Hi Tim,
>>
>> minimal_example_empty was just running an HTTP server that would only
>> return empty replies ;-).  Anyway, the key answer was to try with HTTPS.
>> With that, I could reproduce and fix the issue.
>>
>> It has been fixed in g5e5dbf98..1fe2d0d3 (MHD_VERSION: 0x00096703)
>>
>> I've also added testcases for this.
>>
>> The issue seems minor, but would a new release be useful for you at this
>> time?
>>
>> Happy hacking!
>>
>> Christian
>>
>> On 10/26/19 6:07 PM, Tim Rühsen wrote:
>>> Hi Christian,
>>>
>>> I am just staring at minimal_example_empty.c and wonder what should
>>> happen when I call the executable !? Anyways, I can't easily fill it
>>> with content that works as a reproducer.
>>>
>>> It seems to occur only with HTTPS, HTTP definitely works as expected.
>>>
>>> To reproduce:
>>> git clone https://gitlab.com/gnuwget/wget2.git
>>> cd wget2
>>> ./bootstrap
>>> ./configure
>>> make -j$(nproc)
>>> make -j$(nproc) check -C tests TESTS=test--https-enforce-soft2
>>>
>>> It doesn't come back immediately... so open up a second terminal and
>>>   tail -f tests/test--https-enforce-soft2.log
>>>
>>> You will see the GET request from wget2 and then waiting 10s without
>>> seeing the response. With MHD <= 0.9.66 you'll immediately get the response.
>>>
>>> The MHD server code is in tests/libtest.c.
>>> wget_test_start_server() starts the servers, one for HTTPS and another
>>> one for HTTP (which is likely not relevant here).
>>>
>>> The HTTPS server is started around L733 with
>>>     httpsdaemon = MHD_start_daemon(MHD_USE_SELECT_INTERNALLY | MHD_USE_TLS
>>> #if MHD_VERSION >= 0x00096302
>>>             | MHD_USE_POST_HANDSHAKE_AUTH_SUPPORT
>>> #endif
>>>     ,
>>>     port_num, _check_to_accept, (void *) (ptrdiff_t) SERVER_MODE,
>>> &_answer_to_connection, NULL,
>>>     MHD_OPTION_HTTPS_MEM_KEY, key_pem,
>>>     MHD_OPTION_HTTPS_MEM_CERT, cert_pem,
>>> #ifdef MHD_OPTION_STRICT_FOR_CLIENT
>>>     MHD_OPTION_STRICT_FOR_CLIENT, 1,
>>> #endif
>>>     MHD_OPTION_CONNECTION_MEMORY_LIMIT, (size_t) 1*1024*1024,
>>>     MHD_OPTION_END);
>>>
>>>
>>> The test executes wget2 sends a request to the HTTPS server (as you can
>>> see in the .log file of the test):
>>>
>>> 26.172501.240 GnuTLS init
>>> 26.172501.253 Certificates loaded: 128
>>> 26.172501.253 GnuTLS init done
>>> 26.172501.253 TLS False Start requested
>>> 26.172501.253 ALPN offering h2
>>> 26.172501.253 ALPN offering http/1.1
>>> WARNING: The certificate's (stapled) OCSP status has not been sent
>>> 26.172501.267 TLS False Start: off
>>> 26.172501.267 GnuTLS: Get ALPN: The requested data were not available.
>>> 26.172501.267 Handshake completed
>>> 26.172501.267 established connection localhost
>>> 26.172501.267 # sent 233 bytes:
>>> GET /robots.txt HTTP/1.1
>>> Host: localhost
>>> Accept-Encoding: gzip, deflate, bzip2, xz, lzma, br, zstd, lzip
>>> Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
>>> User-Agent: wget2/1.99.2
>>> Connection: keep-alive
>>>
>>> 26.172501.267 [0] action=2 pending=1 host=0x5606a570abf0
>>> 26.172501.267 ### req 0x7f331c2e9380 pending requests = 1
>>> ###### here the client is waiting for the response for max 10s
>>>
>>>
>>> Hopefully, the above information gives you a chance to track the issue down.
>>>
>>> Regards, Tim
>>>
>>> On 24.10.19 18:01, Christian Grothoff wrote:
>>>> Hi Tim,
>>>>
>>>> Your one-liner just fixes the FTBFS and is itself correct.
>>>> The commit just before that broke everything is the one activating ng0's
>>>> big GSoC work which significantly changes the transmission logic (and
>>>> was thus prepared by many, many other patches before).
>>>>
>>>> I do of course have a hunch as to where the issue might be (0 bytes body
>>>> means no send() for the body, after all. So that might prevent proper
>>>> uncorking, and that might trigger in a test suite that expects low
>>>> latency. But then again, your report says "no reply at all", not just
>>>> +200 ms. So strange.
>>>>
>>>> Anyway, I would really appreciate being able to reproduce this one. You
>>>> could also point me to the wget code, I'd dig there myself, but it would
>>>> likely take a bit longer in that case ;-).
>>>>
>>>> Happy hacking!
>>>>
>>>> Christian
>>>>
>>>> On 10/24/19 4:07 PM, Tim Rühsen wrote:
>>>>> Hi Christian,
>>>>>
>>>>> thanks for the test file - I'll try to copy our code into it.
>>>>>
>>>>> Meanwhile I bisected the git commits...
>>>>>
>>>>> The last functioning is
>>>>> 02a9ae28d64498266869b49b042905946df7ce66 "synt"
>>>>>
>>>>> The next one is not buildable here
>>>>> 379da4ce093bdc957b53b563aa1ae0c7c37c19ac "configure.ac: define a check
>>>>> for HAVE_SENDMSG"
>>>>>
>>>>> And this is the commit that broke our test suite
>>>>> 8ce24c2ae433fdf1ec125211d3622f3c27b56797 "_len -> _size"
>>>>>
>>>>> It's just a one-liner... I hope this helps :-)
>>>>>
>>>>> Regards, Tim
>>>>>
>>>>>
>>>>> On 10/24/19 12:48 PM, Christian Grothoff wrote:
>>>>>> Hi Tim,
>>>>>>
>>>>>> I cannot reproduce this. We do test for empty body in test_get.c, and
>>>>>> that test passes. I also wrote (and now pushed to master) a
>>>>>> minimal_example_empty.c, using both the 200 and 204 status codes. It
>>>>>> works for me on loopback as well as over the network, using both curl
>>>>>> and wget as clients.
>>>>>>
>>>>>> Maybe your issue is about a specific threading mode or something else
>>>>>> you are doing?  Please modify the minimal_example_empty.c to emulate the
>>>>>> situation that is causing the issue.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>> Christian
>>>>>>
>>>>>> On 10/23/19 3:39 PM, Tim Rühsen wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> with 0.9.67 some tests for Wget2 fail which were fine with 0.9.66. This
>>>>>>> is still reproducible with latest master.
>>>>>>>
>>>>>>> As it seems, a response with an empty body doesn't send a response any 
>>>>>>> more.
>>>>>>>
>>>>>>> That means
>>>>>>>
>>>>>>>   response = MHD_create_response_from_buffer(0, (void *) "",
>>>>>>> MHD_RESPMEM_PERSISTENT);
>>>>>>>
>>>>>>> doesn't work, while
>>>>>>>
>>>>>>>   response = MHD_create_response_from_buffer(1, (void *) "x",
>>>>>>> MHD_RESPMEM_PERSISTENT);
>>>>>>>
>>>>>>> still generates a response.
>>>>>>>
>>>>>>> I tested this with MHD_HTTP_BAD_REQUEST, MHD_HTTP_OK and
>>>>>>> MHD_HTTP_NOT_FOUND used for MHD_queue_response().
>>>>>>>
>>>>>>>
>>>>>>> Regards, Tim
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

signature.asc
Description: OpenPGP digital signature