[lwip-devel] httpserver_raw with custom files

[Gawie de Vos]

Hi

First of all, I want to thank and congratulate everybody involved in lwIP
for making such a high quality stack accessible to everyone.

I just want to share my experiences while implementing the httpserver_raw
contribution together with lwIP. My reasons are twofold - (1) to test my
understanding and (2) to help eliminating possible issues. I tried to
"fix" my issues by changing the source as little as possible.

I am using lwIP v1.4.0RC1 and the latest httpserver_raw from CVS on a
STM32F105 Conrtex-M3 with an external Wifi Module (Microchip's ZG2100) via
SPI. My requirement is to serve webpages from internal flash as well as SD
Card, therefore I had to implement my own custom functions to access http
data on the SD Card - enabling the LWIP_HTTPD_CUSTOM_FILES option. I also
have the LWIP_HTTPD_DYNAMIC_HEADERS option set, LWIP_HTTPD_SSI and
LWIP_HTTPD_CGI options set, TCP_MSS set to 512 and TCP_SND_BUF set to (2 *
TCP_MSS).

I have listed my issues below. Thanks again for all your efforts.

Regards,
Gawie


1. There wasn't a call to fs_read_custom() from the fs_read() function
(fs.c). I added it.

2. Change the http_init_file() (httpd.c) function to initialize hs->left
to 0 when a custom file is read. I had to do this in order to read a chunk
of data from the SD Card when http_send_data() is called the first time.

3. Extend HTTP_IS_DATA_VOLATILE(hs) macro to return TCP_WRITE_FLAG_COPY
when dealing with a custom file. I used the (hs)->handle->is_custom_file
flag. I experienced some data corruption when http_write() was called
without TCP_WRITE_FLAG_COPY.

4. Force a tcp send (tcp_output(pcb)) after each call to the
http_send_data() function. I used the return value of http_send_data()
whether or not to force it. I added this to ensure deterministic sending
of tcp packets and therefore helped me when debugging with WireShark. Not
sure if this is needed, but I kept it as it is working very nicely now.

5. The len variable within the http_send_data() function gets updated with
each header string that gets added dynamically. Once all the header
strings has been added and there is file data to be send, the value of len
is set to tcp_sndbuf(pcb). I found that tcp_sndbuf(pcb) sometimes returns
1024 (2*MSS - full buffer available) and in that instances I never saw
that header bytes was sent in WireShark. I don't understand why, and this
may need some more investigation. I "fixed" this by always return after
sending the dynamic headers.

6. I also extend the debug string printed in http_sent() to include the
amount of bytes sent. By doing this I could see that most of the times the
full tcp sendbuf are acked, but sometimes only 512 bytes (MSS) was acked.
This is all fine while the file is being transmitted, but this could
result in the premature closing of a connection when this happens when the
last chunk of the file was sent. For instance (see extract of log file
below), the httpserver sends 1020 bytes (> MSS and < 2*MSS) and the first
512 bytes get acked, the httpserver will now try to read more data from
the file but EOF is reached and then the connection is closed. It is not a
train smash, but we close the connection before the last packet has been
confirmed delivered. I "fixed" this by adding a to_be_acked variable to
the http_state structure. This variable is incremented with the amount of
bytes every time data is requested to be sent and decremented with the
amount of bytes on every ack (http_sent()). The connection is only closed
when EOF is reached and all the bytes have been acked. This change maybe a
little elaborated, but it is doing the job for me.
 

<23:30:05> http_sent 200015e0 1024
<23:30:05> http_send_data: pcb=200015e0 hs=200045e0 left=0
<23:30:05> Trying to read 1024 bytes.
<23:30:05> Read 1024 bytes.
<23:30:05> Trying to send 1024 bytes
<23:30:05> Sent 1024 bytes
<23:30:05> send_data end.
<23:30:05> tcp_output
<23:30:05> http_sent 200015e0 1024
<23:30:05> http_send_data: pcb=200015e0 hs=200045e0 left=0
<23:30:05> Trying to read 1024 bytes.
<23:30:05> Read 1020 bytes.
<23:30:05> Trying to send 1020 bytes
<23:30:05> Sent 1020 bytes
<23:30:05> send_data end.
<23:30:05> tcp_output
<23:30:05> http_sent 200015e0 512
<23:30:05> http_send_data: pcb=200015e0 hs=200045e0 left=0
<23:30:05> Trying to read 1024 bytes.
<23:30:05> End of file.
<23:30:05> Closing connection 200015e0