RE: [lwip-users] tcp_write() errors on snd_queuelen

[Tim Lambrix]

Kieran,

I did change the data I am sending to more human readable data and found that 
all the data is going out in WireShark until the tcp_write returns the error.

I spent a lot of time digging into this and finally found the problem.  I am 
hoping you can help me determine if it is something I am doing wrong or Texas 
Instruments or lwip.

First, below is a portion of the attached log file from lwip running with the 
TCP_QLEN_DEBUG enabled.

        ... [line 3642]
        tcp_enqueue: 37 (after enqueued)
        tcp_enqueue: queuelen: 37
        tcp_enqueue: 38 (after enqueued)
        tcp_receive: queuelen 38 ... 23 (after freeing unacked)
        tcp_receive: queuelen 23 ... 0 (after freeing unacked)
        tcp_enqueue: queuelen: 0
        tcp_enqueue: 1 (after enqueued)
        tcp_enqueue: queuelen: 1
        ...
        ... [line 3764]
        tcp_enqueue: 59 (after enqueued)
        tcp_enqueue: queuelen: 59
        tcp_receive: queuelen 59 ... 35 (after freeing unacked)
        tcp_receive: queuelen 35 ... 11 (after freeing unacked)
        tcp_enqueue: 60 (after enqueued)
        tcp_enqueue: queuelen: 60
        tcp_enqueue: 61 (after enqueued)
        tcp_enqueue: queuelen: 61
        tcp_enqueue: 62 (after enqueued)
        tcp_enqueue: queuelen: 62
        ...
        ... [line 3853]
        tcp_enqueue: queuelen: 102
        tcp_enqueue: 103 (after enqueued)
        pcb->nrtx > 12
        tcp_enqueue: queuelen: 103
        tcp_enqueue: 104 (after enqueued)
        tcp_enqueue: queuelen: 104
        tcp_enqueue: 105 (after enqueued)
        tcp_enqueue: queuelen: 105
        tcp_enqueue: 106 (after enqueued)
        tcp_enqueue: queuelen: 106
        tcp_enqueue: 107 (after enqueued)
        tcp_receive: queuelen 107 ... 95 (after freeing unacked)
        tcp_receive: queuelen 95 ... 71 (after freeing unacked)
        tcp_receive: queuelen 71 ... 48 (after freeing unacked)
        tcp_receive: valid queue length
        tcp_enqueue: queuelen: 48
        tcp_enqueue: pbufs on queue => at least one queue non-empty
        tcp_enqueue: 49 (after enqueued)
        tcp_receive: queuelen 49 ... 48 (after freeing unacked)
        tcp_receive: valid queue length

Normally, we see the tcp_receive prefix take out queues and usually to 0 
(though not always - line 3241 in file).  However, at line 3765, the tcp 
receive interrupt went off during a tcp_write (tcp_enqueue).  We found that in 
the tcp_write, the queue length is read near the beginning of the function into 
a local variable and then stored back into the global variable toward the end 
of the function (see below).  From lwip version 1.3.2 in the tcp_out.c file:

        Line 195          queuelen = pcb->snd_queuelen;

        Line 411          pcb->snd_queuelen = queuelen;

As you can see from line 3765 of the log file, the tcp_receive removed queues 
from the buffer after the tcp_enqueue had read the value to process.  It then 
sets the queue length at the end to the internally modified local value.

So when we get to line 3866, the queue length is incorrect and the value will 
not ever get back to a zero.  If this situation happens enough over time, 
eventually it will reach the TCP_SND_QUEUELEN limit and not function any 
longer. So it explains why I don't see missed packets on WireShark as it is an 
lwip variable that is getting set wrong.  I assume this would affect the number 
of pbufs in use after this point.

It would seem to me that the tcp_enqueue function should only add to the global 
value the number of packets that it uses and not resave the entire value over 
the global to something that could now be old.  Do you agree?  Is there 
something else in the lwipopts.h file that I may not have configured correctly 
that is suppose to prevent this?  I am using a TI Cortex-M3 Stellaris port of 
the lwip code. If you think it is a porting issue, I can try to explain this TI?

Thanks,
Tim


-----Original Message-----
From: address@hidden [mailto:address@hidden On Behalf Of Kieran Mansley
Sent: Monday, March 14, 2011 8:33 AM
To: Mailing list for lwIP users
Subject: RE: [lwip-users] tcp_write() errors on snd_queuelen

I wouldn't suggest manually setting the value to zero - it will likely
confuse lwIP greatly.

Perhaps a good way to see what is going on would be to make sure that
your calls to tcp_write() have identifiable (human readable) data, e.g.
first write is all zeros, second write all ones, and so on.  Then get a
wireshark packet capture to coincide with your tests as described above.
This should show us what is going on the wire, and by deduction what is
queued but not sent.  The packet capture may also reveal why lwIP can't
send any more.  

Kieran


_______________________________________________
lwip-users mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/lwip-users

TestRun2Log.txt
Description: TestRun2Log.txt