[lwip-users] Fast ARP reply causes queueing problem in etharp_query function

[Dave Harper]

I am running lwIP along with FreeRTOS on an embedded microcontroller 
that does periodic checks to a GPS based timeserver located in the same 
room.  Thus the timeserver is able to quickly respond to an ARP request 
(measurements show a consistant range of 740us to 750us). The first 
timeserver check is performed as the system starts up so this causes an 
ARP request to be sent.  In the etharp_query() function, this situation 
is detected, an ARP request is sent and the IP packet is queued for 
later transmission.  Unfortunately, sending the ARP request first and 
THEN queueing the IP packet seems to open up a potential timing 
problem.  What I'm seeing is that occasionally between the time the the 
ARP request is sent and the time the IP packet is actually queued, there 
is an interruption of execution in etharp_query.  Before etharp_query() 
gets to execute again, the timserver is able to respond with the ARP 
reply and this triggers input processing.  As part of the input 
processing update_arp_entry() is called which is where queued packets 
are sent.  Unfortunately, it finds nothing queued at this point so it 
simply returns.  Eventually etharp_query() resumes execution and queues 
the IP packet but at this point it is too late.

One of my debugging tools is a logic analyzer I use to trace the path of 
execution.   This is done by embedding a macro that writes a unique 
value to an I/O port where it is then captured by the logic analyzer.  
What I found was the more macros I embedded to try and trace down the 
source of the problem, the worse it became (I was effectively delaying 
code execution and giving the timeserver more time to respond).  By the 
time I finally figured out what was happening my SNTP request was 
succeeding only about 10% of the time.

I decided to try reversing the order of operations in etharp_query - 
queue the IP packet first and then send the ARP request.  What I found 
was that the SNTP request succeeded every time.  Unfortunately I don't 
have enough of a grasp of full lwIP operation to know if I've only 
appeared to fix a problem but in reality simply opened up other areas 
for failure.  I am running an older version of lwIP (1.3) but I checked 
the latest 1.41 release and etharp_query() operates essentially the same.

Can anyone please weigh in on this?

Thanks,
Dave Harper