[lwip-users] Lwip slow received data speed

[Keith Rubow]

I am attempting to use Lwip 2.0.3 in an embedded system with an 
STM32F427 ARM processor. I am running NO_SYS = 1 mode using the RAW API. 
My ethernet interface is a Wiznet WIZ810MJ module in MAC RAW mode (not 
the ARM on chip ethernet interface, for reasons I will not go into). 
Mostly things seem to be working well. I use one TCP connection, and one 
UDP port.

My problem is very slow data transfer when I attempt to send a fairly 
large amount of data from my PC to the Lwip system. This is done by 
using the Teraterm terminal emulator on my PC, and doing a Send File to 
send a text file via a raw TCP connection. The text file I am sending is 
an Intel HEX file, the contents of which must be programmed into FLASH 
memory in my system. Because the data is being programmed into FLASH 
memory, there are some delays in processing the data due to waiting for 
FLASH program and/or erase cycles to complete. But the slowdown in data 
transfer is MUCH worse than what would be expected from these delays.

My overall architecture is as follows:

The tcp_recv callback simply queues the received data by saving a 
pointer to the PBUF (if there is no data in the queue), or by calling 
pbuf_chain to chain the PBUF to the end of the currently queued data if 
there is already data in the queue.

Another task called in my main loop attempts to transfer data from the 
PBUFs in the received data queue into another buffer (a FIFO buffer used 
by my application). Whenever a PBUF is "emptied" by having all of its 
data transferred into the FIFO, the PBUF is unlinked from the front of 
the queue (by setting its next pointer to NULL, and tot_len=len), and is 
freed by calling pbuf_free. I then call tcp_recved with the len of the 
pbuf I just freed.

Meanwhile, the main data processing task (also called in my main loop) 
is emptying data out of the FIFO, processing the intel HEX data, and 
programming it into FLASH memory. It will also call tcp_output every so 
often to send a "." character back to the user to indicate that data is 
being processed. It will call tcp_output if the call to tcp_write was 
successful.

I am using most of the default options from opt.h. TCP_MSS is 536, 
TCP_WND is 4*TCP_MSS, MEMP_NUM_PBUF is 16. I changed MEM_SIZE to 5000 in 
my lwipopts.h file, thinking that it should be bigger than TCP_WND. That 
didn't seem to help speed up received data.

I did some Wireshark captures to see what was going on. When sending 
data from Teraterm, and writing the received data in to FLASH, 
everything seems go go fine for a little while. The attached file 
(LwipTeratermWritingToFlash.pcapng) shows HEX data being transmitted 
from my computer (192.168.1.9) starting at frame 37. Starting at frame 
45 I start to see TCP Retransmissions from my PC, TCP Dup ACKs from Lwip 
(192.168.1.100), ACKed unseen segment, TCP Out-Of-Order, and other stuff 
that does not look good. Sending the 23062 byte long HEX file took 
11.344 seconds. In spite of all the bad stuff going on, Lwip was able to 
deliver all the data to me, eventually!

For comparison, I modified my system to process the incoming HEX data, 
but bypass the actual writing of data to FLASH. This made the data 
processing go really fast. The attached file 
(LwipTeratermNoWrite.pcapng) shows the same data being sent in 0.530 
seconds. Not a single frame was flagged by Wireshark as anything out of 
the ordinary.

I would have expected the total delay added by the FLASH memory program 
and erase cycles to be no more than 1.5 seconds, so my first test should 
have finished sending the data in about 2 seconds, not 11.344 seconds. 
Note that this test was run with a VERY small HEX file. In the real 
world, the HEX files would be hundreds of times this size, and this sort 
of slowdown would NOT be acceptable.

I think something is going wrong with the TCP data flow control, which I 
believe is handled automatically by the advertised window size. Or maybe 
Lwip is running out of some memory buffers? As I said earlier, 
increasing MEM_SIZE didn't seem to help, and the Wireshark captures seem 
to indicate I have enough PBUFs based on how much data the PC sends 
before it is ACKed. Is there something fundamentally wrong with how I am 
handling the received data? Does my received data callback have to do 
anything special if the received data is queued for later processing? I 
am new to Lwip, and any suggestions would be appreciated.

Keith Rubow

LwipTeratermWritingToFlash.zip
Description: Binary data

LwipTeratermNoWrite.zip
Description: Binary data