Re: [Paparazzi-devel] possible I2C problems halting the main loop, sys_mon module and telemetry settings

[Felix Ruess]

Hi Andre,

I added a similar timeout counter for testing a while ago as well, of course I then couldn't reproduce the problem anymore so I couldn't really test it :-(

Haven't really had time to dig into the issue again since, so your help is very much appreciated!

The question also is how much effort to put into fixing (writing workarounds) for the current driver... since the new driver is apparently basically done..

@Christophe, any comments on this?

Cheers, Felix

On Tue, Feb 21, 2012 at 8:06 PM, Andre Devitt <address@hidden> wrote:

I have recently run into this issue and am in the process of testing a fix.

Basically what can happen is that if a slave holds SDA low the i2c
driver detects this condition and tries to correct with a hard reset.
This can happen in some situations where a slave is confused and
thinks that it should be sending an ACK or if there is a hardware
fault that shorts SDA to ground, or perhaps even if the pullup on the
SDA line has not been installed or falls off :)

The following TI wiki page briefly explains the topic of "External
slave hanging the bus by holding SDA low"
http://processors.wiki.ti.com/index.php/I2C_Tips

Basically what the i2c_hard_reset function does is:

1) Reconfigure the SDA/SCL pins as GPIO so the processor has full
control of the pins (not the i2c peripheral)
2) Toggle SCL high and low until SDA pin is released.
3) generate stop condition, then start
4) reinitialize i2c peripheral

The system will hang if SDA cannot be successfully brought about to a
high condition.

I added in a counter in the while loops to check how many times they
have failed their test condition and just abort the i2c_hard_reset if
it occurs too many times. Tested on my stm32 board and it eliminates
the processor hang.

Of course this still doesn't address some problems if you are running
I2C motor controllers and the bus hangs and aborts you're still
probably in a rough spot :)

Pasted in below the function I tested yesterday, let me know if there
is any input on whether this is the right approach to take for an I2C
bus failure.

Andre.

#define I2C_MAX_RESET_FAIL_COUNT 20

static inline void i2c_hard_reset(struct i2c_periph *p)
{
 uint8_t timeout_fails=0;
 I2C_TypeDef *regs = (I2C_TypeDef *) p->reg_addr;

 I2C_DeInit(p->reg_addr);

 GPIO_InitTypeDef GPIO_InitStructure;
 GPIO_InitStructure.GPIO_Pin = p->scl_pin | p->sda_pin;
 GPIO_InitStructure.GPIO_Speed = GPIO_Speed_10MHz;
 GPIO_InitStructure.GPIO_Mode = GPIO_Mode_Out_OD;
 GPIO_SetBits(GPIOB, p->scl_pin | p->sda_pin);
 GPIO_Init(GPIOB, &GPIO_InitStructure);

 while((GPIO_ReadInputDataBit(GPIOB, p->sda_pin) == Bit_RESET) &&
timeout_fails < I2C_MAX_RESET_FAIL_COUNT) {
   // Raise SCL, wait until SCL is high (in case of clock stretching)
   GPIO_SetBits(GPIOB, p->scl_pin);
   while ((GPIO_ReadInputDataBit(GPIOB, p->scl_pin) == Bit_RESET) &&
timeout_fails < I2C_MAX_RESET_FAIL_COUNT) {
     i2c_delay();
     timeout_fails++;
   }
   i2c_delay();

   // Lower SCL, wait
   GPIO_ResetBits(GPIOB, p->scl_pin);
   i2c_delay();

   // Raise SCL, wait
   GPIO_SetBits(GPIOB, p->scl_pin);
   i2c_delay();
   timeout_fails++;
 }

 // Generate a start condition followed by a stop condition
 GPIO_SetBits(GPIOB, p->scl_pin);
 i2c_delay();
 GPIO_ResetBits(GPIOB, p->sda_pin);
 i2c_delay();
 GPIO_ResetBits(GPIOB, p->sda_pin);
 i2c_delay();

 // Raise both SCL and SDA and wait for SCL high (in case of clock stretching)
 GPIO_SetBits(GPIOB, p->scl_pin | p->sda_pin);
 while (GPIO_ReadInputDataBit(GPIOB, p->scl_pin) == Bit_RESET &&
timeout_fails < I2C_MAX_RESET_FAIL_COUNT) {
   i2c_delay();
   timeout_fails++;
 }

 // Wait for SDA to be high
 while (GPIO_ReadInputDataBit(GPIOB, p->sda_pin) != Bit_SET &&
timeout_fails < I2C_MAX_RESET_FAIL_COUNT)
 {
   i2c_delay();
   timeout_fails++;
 }

 // SCL and SDA should be high at this point, bus should be free
 // Return the GPIO pins to the alternate function
 GPIO_InitStructure.GPIO_Mode = GPIO_Mode_AF_OD;
 GPIO_Init(GPIOB, &GPIO_InitStructure);

 I2C_DeInit(p->reg_addr);

 i2c_apply_config(p);

 if (regs->SR2 & I2C_BUSY) {
   // Reset the I2C block
   I2C_SoftwareResetCmd(p->reg_addr, ENABLE);
   I2C_SoftwareResetCmd(p->reg_addr, DISABLE);

 }
}





On Tue, Feb 21, 2012 at 1:33 PM, Simon Wilks <address@hidden> wrote:
> I am having the same problem too. I had this happen on the Lisa/L board with
> my octocopter using MKK controllers. I have since replaced the Lisa/L with
> the Lisa/M which resolved another issue but after running on the bench for
> about 20 minutes without a problem it suddenly froze when I tried again this
> week. Motors were just idling and I was luckily still bench testing.
>
> @Loic: I experienced the same thing you did with the throttle delay. If you
> search the mailing list for "Strange throttle behavior" you see a suggestion
> from Eduardo on adjusting the gains. After doing this the problem went away.
>
> Regards,
>
> Simon
>
>
> On Tue, Feb 21, 2012 at 7:21 PM, Felix Ruess <address@hidden> wrote:
>>
>> Hi Loic,
>>
>> A too high telemetry rate should not affect other things running, if the
>> message/uart tx buffer is full, they will just be dropped.
>>
>> However: My Lisa/M board also got completely stuck a few times (luckily
>> only while bench testing).
>> Some JTAG debugging showed that it got stuck in a while loop:
>> seemingly after trying to recover from some i2c error on the bus with the
>> mkk controllers:
>> * SPURIOUS_INTERRUPT called
>> * calls abort_and_reset(i2c1)
>> * which calls i2c_hard_reset(i2c1)
>> * inside i2c_hard_reset it got stuck in ```while
>> (GPIO_ReadInputDataBit(GPIOB, p->scl_pin) == Bit_RESET);```
>>
>> Having a while loop here without timeout is bad... but I don't quite
>> understand why it got stuck there... and I can't reproduce it anymore...
>> might have been a hardware problem in my case (since the SCL line never went
>> high again)
>>
>> But we really really need a new I2C driver!
>> Christophe already wrote a new one and it is mostly tested. But there are
>> still some issues with e.g. the aspirin driver (might not be a i2c driver
>> problem per se, maybe initialization timing problem with the gyro?).
>> I have only had problems with the mkk controllers so far (but then that is
>> what I mostly tested), and I get quite a few I2C acknowledge failures on the
>> bus with the mkk controllers...
>>
>> It could be that with more "traffic" on the uart and the accompanying
>> higher load (interrupts) causes i2c errors to occur more often, but that is
>> just a wild guess....
>>
>> Cheers, Felix
>>
>> On Tue, Feb 21, 2012 at 3:45 PM, Loic Drumettaz <address@hidden>
>> wrote:
>>>
>>> Yes, i meant just check. I will add the sys_mon module to my
>>> configuration. Other than delay on the motor, I had also some bugs with my
>>> Lisa M board: the board freezes with all leds on, and the motor stops
>>> (happenned in flight only once!). This seems to happen more often when i set
>>> the telemetry message to "attitude". Is it possible that when the telemetry
>>> rate is too high, the board stops?
>>>
>>> Hi Loic,
>>>
>>>> Thank you for updating the wiki. I had a strange behaviour using the
>>>> int_cmpl_quat: there was some delay in the thrust command when reducing
>>>> throttle. So I went back to int_cmpl_euler but I think i need to do more
>>>> testing with int_cmpl_quat.
>>>
>>>
>>> Commanded thrust has nothing to do with attitude estimation...(unless you
>>> run into lower/upper motor command limits with supervision).
>>>
>>>>
>>>> Maybe this problem was related to the level of vibration I had before
>>>> balancing motoprops...
>>>> About the running frequency of the algorithms: is it possible to make
>>>> sure that the calculation is not too long compared to the sample time? Is it
>>>> possible that the MCU gets overloaded?
>>>
>>>
>>> You can't make sure that everything get's computed in time.. or if you
>>> meant to just check:
>>> The sys_mon module gives you some information about the timing of the
>>> periodic tasks. It will give you a rough average  (over 1 sec) cpu load,
>>> perdiodic time, periodic cycle time and min max of these. So your max should
>>> not be over the periodic time, otherwise in at least one cycle it took
>>> longer to calculate everything and the next one was slightly delayed...
>>>
>>> Cheers, Felix
>>>
>>> _______________________________________________
>>> Paparazzi-devel mailing list
>>> address@hidden
>>> https://lists.nongnu.org/mailman/listinfo/paparazzi-devel
>>>
>>
>>
>> _______________________________________________
>> Paparazzi-devel mailing list
>> address@hidden
>> https://lists.nongnu.org/mailman/listinfo/paparazzi-devel
>>
>
>
> _______________________________________________
> Paparazzi-devel mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/paparazzi-devel
>

_______________________________________________
Paparazzi-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/paparazzi-devel