Bootloader watchdog features

[ccrome]

Currently, when something goes wrong with the application, the board really relies on the ability to double-tap the reset button, or a user button with #294. We have a deeply embedded application, where it would require sending a tech to the field and unscrewing a bunch of screws and pressing buttons and doing some kind of dance to get things to work.

So... any thoughts on how to make the system really foolproof, and virtually never require any button presses of any kind, and always provide the ability get back to the bootloader for hands-off remote system updating?

Here are my thoughts:

In the bootloader, configure the watchdog.

It's possible that a watchdog event is a transient thing, and the system will continue as normal if allowed to boot back to the application. Not great, but possible. So, keep a watchdog trigger count. Watchdog trigger count (WTC) is reset by the application when it's in a known good state.

WTC = 0 or 1 : boot to application
WTC >= 2: erase app and stay in bootloader

That way, if things go sideways, the bootlaoder will always come back up automatically.

Another option would be to load some known-good backup firmware on WTC >=2.

[dhalbert]

Which chip are you using? Several chips have hardware-settable watchdog timers, which are set in the fuses. For instance the SAMD21 and SAMD51 have such hardware watchdog timers. There is an enable bit in the fuses, and another set of bits to set the watchdog interval.

[ccrome]

My assumption was that pretty much every chip has some kind of watchdog. I'm using the i.MX RT parts, which have between two and three watchdogs :-).

So the hardware is definitely there for many if not most chips. The question is really about policies and methods for deeply embedded systems where there is no physical access to the board.

[hathach]

Bootloader won't play any part in watchdog, application should start the watchdog and reset into bootloader should there is any issue.

[hathach]

IMO, bootloader should not start any watchdog, since

1. application shouldn't got reset by peripheral not started by itself
2. application can easily start its own watchdog and has its own logic to when reset to app or bootloader.

Also the interrupt vector is already transferred to application, bootloader watchdog code won't triggered to run anyway. Let me know if you have any scenario that application watchdog can't handle.

[ccrome]

Here's my actual scenario. Perhaps I'm being paranoid, but I think it's not crazy:

we deploy 1000 devices into the field. These are embedded devices, with zero access to any buttons, and no easy way to power cycle the board or affect a POR reset/boot pin manipulation. We do a firmware update that fails. Let's say, it does something like has the wrong load address or something like that. Bottom line is that the application never actually gets started. Of course, that should never happen, but I've definitely seen weirder accidents happen.

Now, we've got 1000 device in the field across the globe and we need to send a technician out to literally disassemble and do something to the PCB -- double-tap the reset button, or something like that. The price tag for that kind of operation could easily be in the tens of thousands of dollars.

Since the app never gets a chance to start the watchdog, we seem to be completely wedged and need to get physical screwdrivers into the field.

As for the vectors, they just don't matter at that point -- the thing needs a reset to get back to life, which is why I think the watchdog at the bootlaoder stage is really what I need in my application. I totally understand if it's not something that you want to integrate, but I'm implementing it now because I think we need it, unless you come up with some better mechanism :-)

[hathach]

As for the vectors, they just don't matter at that point -- the thing needs a reset to get back to life, which is why I think the watchdog at the bootlaoder stage is really what I need in my application. I totally understand if it's not something that you want to integrate, but I'm implementing it now because I think we need it, unless you come up with some better mechanism :-)

I see, I miss your point on using WTC as hint for bootloader. We won't implement watchdog since it will force an requirement to clear it on existing apps. Another mechanism I can think of is using firmware crc, though where to store crec also require boot-app agreement.

For you application, adding board_teardown_post() could help, you could implement that to enable watchdog for your specific board. The other works is having some hook for custom board_app_valid() so that you could check the WTC and return value that force DFU mode.

[ccrome]

I've implemented a watchdog example here.

Is there a place for some sort of examples or show-and-tell features in the tinyuf2 tree? Assuming there's no desire to put the watchdog features into the main code, it could still be handy to have a place for examples.

[hathach]

since it is board specific code, you can simply submit an PR to add your board hw as example for this watchdog implementation.