After solving issue with fragmented 3G packets I ran into some performance related problems. My application consists of STM32 processor and lwIP running on FreeRTOS. There is http server thread which responsibility is to read page contents from flash and fill it with dynamic data. On local network everything works good. But tests on a 3G network showed that sometimes page doesn't load because of a connection reset error. Instead of a regular page a "Connection has been reset" error page is showed. Sometimes it affects one or more tables which are loaded from separate html files, sometimes it affects whole page.
Further investigation revealed that problem is probably caused by limited memory for memp objects and pbufs. At some point there is not enough space on the heap to allocate one of memp objects. I allocate memp and pbufs on the heap shared with FreeRTOS, heap size is 64kB. When tracking free heap space in a real time I have seen that in idle state there is more than 55kB free. When I send single page request, free heap amount decreases by about 10kB then returns to idle state. When I send fast continuous requests (with F5 spamming) free heap size decreases by about 10-20kB then stays for a while on that level and then goes down to about 10-15kB. When free heap is that low, STM responds with RST flag in TCP packet. After few seconds free memory amount goes up again and http server responds properly. I tried also with static memp pools and I didn't get good results either.
There is still some room for improvement with these memory allocation issues. And we have taken some effort to work on a connection reset problem. If those problems occurs only in a high load situations, we will live with that. The thing is that from time to time it happens also during regular operation, mostly just after long time of inactivity. I dont know if its another instance of the same problem, or something completely different.