RE: [lwip-users] pbuf misuse or snd_queuelen corruption.

[Andrew Foster]

Kieran,                                                                  
               
                                                                         
               
Thanks for the response. As a preface, I appreciate your input on this 
issue. It is not easy to jump into someone's code space and try to 
determine were they've wronged. I've tried to remove code that wasn't 
important to identifying the issue in a hope to streamline your 
evaluation. 

Please see the following:
                                                                         
               
Thread 1: low_level_input                                                
               
Description: Infinite loop waiting for a semaphore signal from the 
Ethernet receive ISR.
Pseudo code: On semaphore signal,                                        
                                   
                                                                         
               
  // We allocate a pbuf chain of pbufs from the pool.                    
               
  p = pbuf_alloc(PBUF_RAW, len, PBUF_POOL);                              
               
                                                                         
               
  // Setup pointer to the Ethernet Header.                               
               
  ethhdr = p->payload;                                                   
               
                                                                         
               
  // Determine the type of packet (IP or ARP) and process accordingly.   
               
  switch (htons(ethhdr->type))                                           
     
  {                                                                      
                                                                         
     
    case ETHTYPE_IP:                                                     
   
        // Update the ARP table.                                         
   
        etharp_ip_input(s_pxNetIf, p);                                   
   
                                                                         
   
        // Skip the Ethernet header.                                     
   
        pbuf_header(p, -((s16_t)sizeof(struct eth_hdr)));                
   
                                                                         
   
        // Pass the packet to the network layer. Where
        // s_pxNetIF->input =  tcpip_input                     
        // err_t tcpip_input(struct pbuf p, struct netif *inp)           
   
        s_pxNetIf->input(p, s_pxNetIf);                                  
   
        break;                                                           
   
                                                                         
   
    case ETHTYPE_ARP:                                                    
   
        // pass the packet to the ARP layer.                             
   
        etharp_arp_input(s_pxNetIf, ethernetif->ethaddr, p);             
   
        break;                                                           
                                                                    
                                                                         
   
    case ETHTYPE_RARP:                                                   
   
        // pass the packet to the RARP layer.                            
   
        etharp_rarp_input(s_pxNetIf, ethernetif->ethaddr, p);            
   
        break;                                                           
   
                                                                         
   
    default:                                                             
   
        // Ignore the packet if it is not IP or ARP.                     
   
        pbuf_free(p);                                                    
   
        p = NULL;                                                        
   
        break;                                                           
   
  }  

Thread 2: tcpip_thread
Description:  The main lwIP thread. This thread has exclusive access to 
lwIP core functions (unless access to them is not locked). Other threads 
communicate with this thread using message boxes.

I have not made any modifications to the tcpip_thread provided in the 
lwIP stack.


Application Layer Description:

For all TCP communications, I'm using the raw api. I've tried to 
condense the application functions so that it focuses on how I use the 
lwIP TCP API. There are probably a few application specific trappings 
left over but I've tried my best to omit things that weren't lwIP 
specific.

The overall model is as follows, with pseudo code included:

I use all the provided callbacks (sent,recv,err,poll) to interface to 
the lwIP TCP/IP stack. After making a connection, I assign the 
appropriate callbacks and timers. 

The sent callback has no interface with the lwIP api.

The recv callback queues all the pbuf allocations to static ram for 
later processing. If an error is detected, or a NULL pbuf is passed in, 
the previously stored pbufs are freed and the connection is terminated.

The err function will free all associated pbufs and terminate the 
connection.

The poll callback processes the static ram pbuf references and extracts 
the buffered data. Once an entire chain is received, the tcp_recved 
function is called and the pbufs are freed. If data is ready to be sent, 
the poll function will check the available space in the tcp window and 
segment count before sending. After calling tcp_write, the application 
attempts to flush the tcp_output by calling tcp_output.


Application Layer Psuedo Code:

tcp_arg(pcb, NULL);
tcp_sent(pcb, TelnetSent);
tcp_recv(pcb, TelnetReceive);
tcp_err(pcb, TelnetError);
tcp_poll(pcb, TelnetHandler, 1);

Function: TelnetSent
Description: This function is called when the lwIP TCP/IP stack has 
received an acknowledgment for data that has been transmitted.

Psuedo code:
// No lwIP api calls in function body.


Function: TelnetReceive
Description: This function is called when the lwIP TCP/IP stack has an 
incoming packet to be processed.

Psuedo code:
TelnetReceive(void *arg, struct tcp_pcb *pcb, struct pbuf *p, err_t err)
{
    SYS_ARCH_DECL_PROTECT(lev);

    // Place the incoming packet onto the queue if there is space.
    if((err == ERR_OK) && (p != NULL))
    {
        // This should be done in a protected/critical section.
        SYS_ARCH_PROTECT(lev);


        // Do we have space in the queue?
        iNextWrite = ((pState->iBufQWrite + 1) % PBUF_Q_SIZE);
        if(iNextWrite == pState->iBufQRead)
        {
            // The queue is full - discard the pbuf and return since we 
can't
            // handle it just now.

            // Restore previous level of protection.
            SYS_ARCH_UNPROTECT(lev);

            // Free up the pbuf.  Note that we don't acknowledge receipt 
of
            // the data since we want it to be retransmitted later.
            pbuf_free(p);
        }
        else
        {
            // Place the pbuf in the circular queue.
            pState->pBufQ[pState->iBufQWrite] = p;

            // Increment the queue write index.
            pState->iBufQWrite = iNextWrite;

            // Restore previous level of protection.
            SYS_ARCH_UNPROTECT(lev);
        }
    } // If a null packet is passed in, close the connection.
    else if((err == ERR_OK) && (p == NULL))
    {
        // Clear out all of the TCP callbacks.
        tcp_arg(pcb, NULL);
        tcp_sent(pcb, NULL);
        tcp_recv(pcb, NULL);
        tcp_err(pcb, NULL);
        tcp_poll(pcb, NULL, 1);

        // Close the TCP connection.
        tcp_close(pcb);
        
        // Clear out any pbufs associated with this session.
        SYS_ARCH_DECL_PROTECT(lev);

        // This should be done in a protected/critical section.
        SYS_ARCH_PROTECT(lev);

        // Pop a pbuf off of the rx queue, if one is available, and we 
are
        // not already processing a pbuf.
        if(pState->pBufHead != NULL)
        {
            pbuf_free(pState->pBufHead);
        }

        while(pState->iBufQRead != pState->iBufQWrite)
        {
            pbuf_free(pState->pBufQ[pState->iBufQRead]);
            pState->iBufQRead = ((pState->iBufQRead + 1) % PBUF_Q_SIZE);
        }
        // Restore previous level of protection.
        SYS_ARCH_UNPROTECT(lev);

        // Clear out the telnet session PCB.
        pState->pConnectPCB = NULL;      
    }

    //
    // Return okay.
    //
    return(ERR_OK);
}


Function: TelnetError
Description: This function is called when the lwIP TCP/IP stack has 
detected an error. The connection is no longer valid.

Psuedo code:
TelnetError(void *arg, err_t err)
{
    SYS_ARCH_DECL_PROTECT(lev);

    // This should be done in a protected/critical section.
    SYS_ARCH_PROTECT(lev);

    // Pop a pbuf off of the rx queue, if one is available, and we are
    // not already processing a pbuf.
    if(pState->pBufHead != NULL)
    {
        pbuf_free(pState->pBufHead);
    }

    while(pState->iBufQRead != pState->iBufQWrite)
    {
        pbuf_free(pState->pBufQ[pState->iBufQRead]);
        pState->iBufQRead = ((pState->iBufQRead + 1) % PBUF_Q_SIZE);
    }
    // Restore previous level of protection.
    SYS_ARCH_UNPROTECT(lev);
}


Function: TelnetHandler
Description: This function is called periodically from the lwIP timer 
thread context. This function will handle transferring data between the 
UART and the telnet sockets. 

Psuedo code:
TelnetHandler(void)
{
    SYS_ARCH_DECL_PROTECT(lev);

    // Pop a pbuf off of the rx queue, if one is available, and we are
    // not already processing a pbuf.
    if(pState->pBufHead == NULL)
    {
        if(pState->iBufQRead != pState->iBufQWrite)
        {
            SYS_ARCH_PROTECT(lev);
            pState->pBufHead = pState->pBufQ[pState->iBufQRead];
            pState->iBufQRead =
                ((pState->iBufQRead + 1) % PBUF_Q_SIZE);
            pState->pBufCurrent = pState->pBufHead;
            pState->ulBufIndex = 0;
            SYS_ARCH_UNPROTECT(lev);
        }
    }

    // If there is no packet to be processed, break out of the loop.
    if(pState->pBufHead == NULL)
    {
        break;
    }

    // Setup the data pointer for the current buffer.    
    // process all the rx data...
    // psuedo cdoe ommited (no lwIP API calls)

    // Check to see if we are at the end of the chain.  If so,
    // acknowledge this data as being consumed to open up the TCP
    // window.
    if(pState->pBufCurrent == NULL )
    {
        tcp_recved(pState->pConnectPCB, pState->pBufHead->tot_len);
        pbuf_free(pState->pBufHead);
    }

    // Flush the TCP output buffer, in the event that data was
    // queued up by processing the incoming packet.
    tcp_output(pState->pConnectPCB);

    // Try to send any data we have ready to go out.
    if( tcp_sndbuf(pState->pConnectPCB) &&
        (pState->pConnectPCB->snd_queuelen < TCP_SND_QUEUELEN) )
    {
        // Here, we have data, and we have space.  Set the total amount
        // of data we will process to the lesser of data available or
        // space available.

        if ( tcp_sndbuf(pState->pConnectPCB) >= len &&
             0 != len )
        {
            tcp_write_error = tcp_write(pState->pConnectPCB, rsp_temp, 
rsp_index, 1);

            if ( ERR_OK != tcp_write_error )
            {
                ASSERT();
            }
        }
    }
    // Flush the data that has been written into the TCP output buffer.  
      
    tcp_output(pState->pConnectPCB);
}


Thanks,

Andrew 

-----Original Message-----
From: Kieran Mansley [mailto:address@hidden 
Sent: Monday, October 18, 2010 07:52
To: Mailing list for lwIP users
Subject: Re: [lwip-users] pbuf misuse or snd_queuelen corruption.

On Thu, 2010-10-14 at 17:42 -0400, Andrew Foster wrote:
> I find myself with circular references

This isn't much help as you already suspect the threading model but
circular references in packet buffers are almost always down to multiple
threads accessing a packet queue at the same time.  Can you give more
details about how exactly received packets are passed to the stack, and
which API your application is using?

Kieran


_______________________________________________
lwip-users mailing list
address@hidden
http://lists.nongnu.org/mailman/listinfo/lwip-users