[lwip-users] About tcp_write() and TCP_SND_QUEUELEN

[R. Diez]

Hi all:

I am having performance problems with TCP and I wanted to understand what is 
going on. It could be the Nagle algorithm.

My devices exchange many small to medium-sized packets. Ideally, I would use TCP_CORK in the main loop to handle output data efficiently. 
But such an option is apparently not available in lwIP.

As a first step, I'm trying to understand how tcp_write() and tcp_output() 
work, especially regarding TCP_SND_QUEUELEN.

Say my application has received 2 complete request data packets and wants to 
send 2 response data packets in a single go.

The first data packet is 20 Bytes long and gets written with tcp_write( 
TCP_WRITE_FLAG_COPY ) without any problems.

Say that lwIP allocates internally a 30-Byte pbuf. 20 bytes are now occupied 
and 10 remain free.

The second data packet is 20 Bytes long too.

My application calls tcp_sndbuf() and gets 1024 bytes back. There is also plenty of room (theoretically). So it calls tcp_write() again. 
However, the application gets ERR_MEM back.

That probably means that the TCP_SND_QUEUELEN limit has been reached, right? 
The application should try again later.

Unfortunately, "later" is not well defined. If I want to send the packets as soon as possible, I could constantly poll, which would mean 
100% CPU load. Or is there a better way to get a notification when the Tx buffer has more room available?

I am guessing that tcp_write() uses an "all or nothing" approach. In the case above, it could have written 10 bytes now, filling up the last 
pbuf. The other 10 bytes could be sent later on.

Is that correct? If so, one could argue that the tcp_write() interface is not quite efficient with regards to buffer management. Or is there 
a way to find out how much data I could write without triggering an ERR_MEM because of TCP_SND_QUEUELEN?

Disabling the Naggle algorithm makes tcp_pbuf_prealloc() allocate one pbuf per tcp_write(), or so I believe. But I have too many small 
packets. Disabling Nagle and raising TCP_SND_QUEUELEN would have a negative impact overall.

Yet another question: If the application could not write the second packet, it does not make any sense to wait any longer before sending the 
first packet out over Ethernet, so Nagle is actually getting in the way. I guess calling tcp_output() does not make any difference either. 
Is there a way to say "send now without waiting for Nagle?".

Best regards,
  rdiez