[lwip-users] Dying listener sockets

[Tamas Somogyi]

Hi,

I experienced the problem that listener sockets don't accept any more incoming 
connection after some time.

I have a server application that listens on several (8-10) ports, and a test 
client that tries to connect to those server ports, 3-6 connections to the same 
port. If a connection established, after some (random 0.1-10s) time the client 
closes the connection and tries again in endless loop. Please find the 
pseudo-code below.

The problem comes after ~50 reconnections (0.5-2min): one or more listener 
sockets don't accept any more connection, so those ports seem dead (client 
sends SYN, but no reply from the server). As time passes, even more ports dies, 
so I end up with 0-3 ports alive.

Some observations:
- I got the above result both on lwip 1.3.0 and on 1.3.2.
- I can reproduce both on Windows and on our target platform (TI DSP).
- It seems that if tune my threads (sleeping a bit between two loops, using 
thread pools, etc.), the problem comes later or earlier (50<-->150-200 
reconnections), so I suspect that some resources may be not protected in lwip 
from concurrent access. Note that I don't use the same socket in different 
threads as you can see in the code below.
- MEMP_NUM_TCP_PCB=32, MEMP_NUM_TCP_PCB_LISTEN=16 - so I have more incoming 
connections than 32, maybe that's the problem.
- There's no error message from the sockets, they are created, closed, etc. 
properly.
- I tried to close and re-create the listener socket after accepting an 
incoming socket, but got the same result. This indicates that the problem is 
probably not in sockets, but in an underlying layer.
- It seems that the problem comes when some sockets are connected, but the 
client still tries to create new connections to the same port. I tried to 
change the backlog in lwip_listen from 2 to 1..20, but still got the problem.
- This kind of usage is not realistic for my target application, but 
unfortunately I experience some dead ports after 12-24h run during "normal" 
usage, this test just brings forward and highlights the problem.
- The dead listeners are stuck in sys_arch_mbox_fetch() waiting for message, 
but tcpip_thread runs happily processing incoming packets.

Can you point me where the problem is - perhaps in my code?
Or if it should work, then has somebody ever tried to do such stress test on 
the sockets? 

Thanks,
Tamas



*** SERVER: ***

listener_thread(port)
{
        //create listening socket
        my_listener_socket = lwip_socket(AF_INET, SOCK_STREAM, 0);

        struct sockaddr_in socket_addr;
        memset(&socket_addr, 0, sizeof(sockaddr_in));
        socket_addr.sin_len = sizeof(sockaddr_in);
        socket_addr.sin_family = AF_INET;
        socket_addr.sin_port = htons(port);
        socket_addr.sin_addr.s_addr = INADDR_ANY;
        lwip_bind(my_listener_socket, socket_addr, sizeof(sockaddr_in));

        lwip_listen(my_listener_socket, 2);

        //wait incoming connections
        while (true) {
                sockaddr sa; socklen_t sa_size = sizeof(sa);
                incoming_socket = lwip_accept(my_listener_socket, &sa, 
&sa_size);

                //create thread for the new incoming connection
                thread_create(connection_thread(incoming_socket));
        }
}

connection_thread(new_socket)
{
        //read data until closure
        while (true) {
                //wait for bytes received - non-blocking call
                bytes_received = lwip_recv(new_socket, buf, sizeof(buf), 
MSG_DONTWAIT);

                //check for shutdown
                if (bytes_received <= 0) {
                        break;
                }
        }

        //close
        lwip_close(new_socket);
}

server_main()
{
        for (i = 1 to NUM_PORTS) { //NUM_PORTS=8
                thread_create(listener_thread(port[i]));
        }
}


*** CLIENT: *** (non-lwip, using windows sockets)

client_thread(port)
{
        //loop forever
        while (true) {
                thread_sleep(random 0.1-10s);
                connect(port);
                thread_sleep(random 0.1-10s);
                disconnect(port);
        }
}

client_main()
{
        for (i = 1 to NUM_PORTS) {
                for (j = 1 to NUM_CONN_PER_PORT) { //NUM_CONN_PER_PORT=3-6
                        thread_create(client_thread(port[NUM_PORTS]));
                }
        }
}