Endless unresponsive loop after ZST39 upgrade to firmware 1.30 (SDK 7.21.3)

[vladm]

Updated zst39 stick to the latest FW 1.30 (SDK 7.21.3) using HA, after that stick boots, does initial network detection and then enters endless 'unresponsive' loop.

Downgraded FW to 1.20 (based on SDK 7.19.3) using Simplicity studio and everything is back to normal.

attached log from docker, please let me know if that's hard to interpret, I can upgrade it one more time and try to log-to-file instead.

also someone else reported same thing here:
https://community.home-assistant.io/t/zooz-zst39-lr-800-firmware-7-21-3-fail/733405

Software versions

zwave-js-ui: 9.12.0.96eeb76
zwave-js: 12.5.6

Device information

Manufacturer: zooz
Model name: zst39
Node ID: 1

Upload Logfile

zw.zip

[zwave-js-bot]

👋 Hey @vladm!

It looks like you attached a logfile, but its filename doesn't look like it a driver log that came from Z-Wave JS.

Please double-check that you uploaded the correct logfile. If you did, disregard this comment.

As a reminder, here's how to create one:

• Z-Wave JS UI
• Home Assistant Z-Wave Integration
• ioBroker.zwave2 Adapter

[AlCalzone]

I don't see the issue happen in the driver logs, only the application logs which don't include the necessary information to diagnose it.

Please unplug and re-plug the stick and include a new driver log which should hopefully capture what you described.

[vladm]

@AlCalzone thank you for your response.

I did update zst39 back to the new 7.21.3, though this time I used Silicon lab's pc controller software from simplicity studio. What's interesting is that after this update controller works OK, ZW JS exhibits no such issues as it had after I used HA to update FW last time. I wonder, if the problem is rather with the update process via Home Assistant and I ended up with corrupted controller last time?

Unrelated, even though controller is at SDK ver 7.21.3, the UI still complains about 7.19.3...

[AlCalzone]

I also tested the update to 7.21.3 for Zooz and I had no issues at all using Z-Wave JS. Not sure what happened in your case.

The warning needs a revision, thanks for the hint. It's still relevant though - 7.21.3 does not fully fix all issues.

[Botched1]

Similar issue here.

Was on 1.1 (7.18.3) and working, but since I was getting probably 30+ "jammed" events a day I thought I would upgrade to see if it helped at all (although I knew it wasn't "fixed" in 7.21.3).

So I updated to 1.3 (7/21/3) in zwave js ui and now I just get non-stop errors in the logs.

I can't seem to downgrade/install any of the gbl in PC Controller or in Zwave JS UI in bootloader only mode either, so probably bricked my device.

Such is life. I have another ZST39 on the way, and have NVM backups, so hopefully I can get a new stick working without re-pairing 94 devices......

[AlCalzone]

I can't seem to downgrade/install any of the gbl in PC Controller or in Zwave JS UI in bootloader only mode either, so probably bricked my device.

AFAIK downgrading isn't possible, unless you have an unofficial bootloader or Zooz built this functionality into theirs.

[pletch]

Updated zst39 stick to the latest FW 1.30 (SDK 7.21.3) using HA, after that stick boots, does initial network detection and then enters endless 'unresponsive' loop.

Downgraded FW to 1.20 (based on SDK 7.19.3) using Simplicity studio and everything is back to normal.

I had the exact same experience as you. Previously running 1.20 without issue for at least 6 months and foolishly decided to upgrade my stick a few days ago via zwave-js-ui interface. I started sporadically experiencing same problem in an LXC container on Proxmox. I was able to restore the ZST39 to normal operation only by shutting down the container and physically disconnecting / reconnecting the stick. I tried requesting soft reset and/or simply restarting the container but these were ineffective.

I downgraded to 7.19.3 via zwave-js-ui interface last night and am hoping all is well again (though I had to manually reset RF region back to USA).

[vladm]

@AlCalzone some update -

I was playing in ZWave-JS UI with rebuilding routes and sending pings to nodes, and suddenly it failed (when I was checking node 044 - which is an old, non zwave plus node with s0 security) and, as others indicated above, controller no longer responsive. Restarting zwave-js container didnt help, I had to unplug the stick to get it back online.

I captured log, see attached:
zwavejs_2024-05-29.zip

[AlCalzone]

No I don't think so. We can't do anything about it anyways.

[exciton]

Why's that? It's definitely a firmware bug?

[AlCalzone]

Yes and it has been plaguing people for months now.

[Botched1]

Yes and it has been plaguing people for months now.

If only there were members of the Zwave Alliance, or those that had an 'in' at SiLabs, that could help push getting this fixed. :)

(I'm joking)

I switched from a ZST39 back to a UZB-7 yesterday, and am seeing about 1/10th the number of jammed alerts that I used to (both sticks on the same 7.18.3 SDK). So that's nice! Although my noise floors and device RSSI are measurably worse on the UZB-7 than they were on the ZST39 - so there is no "free lunch" as they say. 6-7 dB worse on noise floor/background RSSI, for example.

[vladm]

Yes and it has been plaguing people for months now.

I dont have much problems when on 7.19.3, running months..
though 7.21.3 locks pretty fast - matter of minutes or hours. so it's definitely something new.

[exciton]

Got it. Maybe we should put a big warning somewhere telling people not to upgrade to this version? Is there a reliable way to downgrade? I had thought this was prevented by the bootloader but have seen comments indicating it can be done.

…

On Thu, 30 May 2024, 11:39 am Botched1, ***@***.***> wrote: Yes and it has been plaguing people for months now. If only there were members of the Zwave Alliance that could help push getting this fixed. :) (I'm joking) — Reply to this email directly, view it on GitHub <#6874 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AALST5GP7D6XZQRTYEBIMS3ZE5W7VAVCNFSM6AAAAABIN4BE2CVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TMMBZGI2DG> . You are receiving this because you commented.Message ID: ***@***.***>

[pletch]

*** usually ***

My ZST39 got bricked in the process and no new firmware can be flashed now.

Is the bootloader not coming up at all? Have you tried connecting via serial and uploading firmware image via bootloader receive mode?

See here: https://community.silabs.com/s/article/z-wave-700-programming-uzb7-controller-stick?language=en_US

[Botched1]

I haven't tried via serial yet, but plan to when I get time. It comes up enough that I'm hopeful that will work.

So bricked was the wrong term to use. I guess I should have said it will no longer flash via USB in PC Controller (gives an XMODEM error) or Z-Wave js (gives a different error).

[exciton]

I was able to downgrade via zwave-js-ui. Got an error at the end of flashing but it worked fine.

[AlCalzone]

So bricked was the wrong term to use. I guess I should have said it will no longer flash via USB in PC Controller

Happy to look at some logs to see what's going on there. Also, have you tried the "Bootloader only" setting in Z-Wave JS UI? This is meant to re-flash a stick that's stuck in bootloader mode.

[xantonin]

I haven't tried via serial yet, but plan to when I get time. It comes up enough that I'm hopeful that will work.

So bricked was the wrong term to use. I guess I should have said it will no longer flash via USB in PC Controller (gives an XMODEM error) or Z-Wave js (gives a different error).

I was able to unbrick mine via these directions: https://www.reddit.com/r/zwave/comments/15wjmw2/comment/kxsef7a/

[andronat48]

Just a report.

I upgraded ZST39 from 7.19.3 to 7.21.3 via ZwaveJS UI. Infinite controller unresponsive loop. Any kind of firmware update/downgrade fails with error 0x18 (Simplicity Studio, ZwaveJS)

I restored NVM backup to a spare ZST10 which bricked it completely (no terminal).

Then I bought another ZST39, flashed 7.19.3 via Simplicity Studio and restored NVM backup via ZwaveJS UI. After a few loops it seems to work. 800LR devices did not restore.

It seems like upgrading to 7.21.3 via ZwaveJS UI has a chance to brick the stick in a really weird way...

[andronat48]

A quick update on the ZST39.

Things got so bad that my Ring alarm (also based on z-wave) stopped responding to keypads... I narrowed the issues to one particular node (a Zooz ZEN72, I have several of them). The controller often would go unresponsive after communicating with this node.

I could not exclude it no matter what I tried, and could not Remove Failed after factory reset of the node (both zwavejs and simplicity studio). Something related to the node in the failed list in zwavejs and NOP timeout in the studio.

I edited the NVM backup json, removed the node, reset the controller, reflashed 7.19.3, reset again, restored NVM via zwavejs, paired the node under new ID. I also disabled controller recovery

Now it looks like the network is very stable.... No issues whatsoever

P.S. I got another ZST39, will try to copy NVM and upgrade to .21 via studio over the weekend

[AlCalzone]

assuming you mean ZST10 that got completely botched

I meant that, yes. The log looks like no response whatsoever. No bootloader, no serial API. I assume you just plugged it in to the separate PC?
If so, I have no clue what could have caused this, other than you somehow managed to flash an incompatible bootloader without the old one detecting that.

[exciton]

I was able to downgrade to 7.18.3 (heard on HA forums that it's even more stable than 7.19.3), however NVM restore complains of unsupported NVM format. Anyone know a way around this.
I have things going now on 7.19.3 but I figured I'd try restoring on a spare stick with 7.18.3 and see if it's any more stable.

[xantonin]

Sorry I don't mean to spam all these threads but I bricked mine too via update in HA and found this post and was able to unbrick it, hope it helps: https://www.reddit.com/r/zwave/comments/15wjmw2/comment/kxsef7a/

[alexruffell]

What @andronat48 describes appears to be exactly what happened to me resulting in an unrecoverable ZST39. Either I have real back luck using zwave js to update my stick firmware, or there is something off with my system causing it to fail. This has happened to me on other sticks too, but using the cli flash tool fixed it (was stuck in bootloader mode), but not this latest time around.

[memesalot]

Updated zst39 stick to the latest FW 1.30 (SDK 7.21.3) using HA, after that stick boots, does initial network detection and then enters endless 'unresponsive' loop.

Downgraded FW to 1.20 (based on SDK 7.19.3) using Simplicity studio and everything is back to normal.

attached log from docker, please let me know if that's hard to interpret, I can upgrade it one more time and try to log-to-file instead.

also someone else reported same thing here: https://community.home-assistant.io/t/zooz-zst39-lr-800-firmware-7-21-3-fail/733405

Software versions

zwave-js-ui: 9.12.0.96eeb76 zwave-js: 12.5.6

Device information

Manufacturer: zooz Model name: zst39 Node ID: 1

Upload Logfile

zw.zip

Do you have some instructions on how to downgrade firmware versions? I tried many times and had no luck.

[dwaleke]

I told it to perform the downgrade via OTW update and it crashed. Then, I told ZWJS to load in bootloader mode and reperformed the downgrade.

It never left 0%, but it still updated and restarted the controller.

Thank you for this. I am able to get ZWJS to go into bootloader mode but I get the same file error 0x18 as I do everywhere else I attempt to flash older firmware.

z-ui_2024-06-06.log

[ansman]

I had this error on the 1.30 firmware. Downgrading to 1.20 worked fine from Home Assistant but did not resolve the issue. I had to hard reset it and re-pair all devices but it's working well now.

My best guess is that it due to a few dead devices in my network as it seemed to hang after attempting to ping them.

After downgrading from 1.30 --> 1.20, I had to go in through zwave-js-ui configuration and manually change the frequency to the right setting (US in my case). All of the nodes initially appeared dead until I did this.

To be clear it wasn't all devices that were acting dead, only 4 which were actually disconnected

[memesalot]

des initially appeared dead until I did this.

After downgrading from 1.30 --> 1.20, I had to go in through zwave-js-ui configuration and manually change the frequency to the right setting (US in my case). All of the nodes initially appeared dead until I did this.

How is everyone able to downgrade? ZWave JS UI gives an error if you attempt to flash an older version. Simplicity Studio also errors out.

Seems the stick needs to be in bootloader mode to do it via terminal/xmodem. What is the trick to get the stick to boot into bootloader mode so we can flash an older firmware version?

Did you ever find out how to do this? I have no idea how to get it into bootloader mode on windows so I can try the xmodem way

[dwaleke]

Did you ever find out how to do this? I have no idea how to get it into bootloader mode on windows so I can try the xmodem way

ZWave JS UI will do it (posted log above), but I always get an error attempting to flash the stick. I see many posts from people flashing older versions but it is not clear to me how they are doing it. ZWave JS UI and Simplicity Studio both error out immediately.

[AlCalzone]

It depends on your bootloader. Apparently some of the sticks have one that allow downgrading, I guess the 2nd generation of sticks or so.

[droans]

Haven't seen anyone mention it, but the Z-Wave SDK 7.21.3 release notes mention an issue which can cause 700/800 controllers to lock up. Not sure if this is the cause or not, but I'd guess so.

[AlCalzone]

There are two "lockup" issues in the firmware.

1. The controller is still responsive but refuses to transmit (this is in the release notes)
2. The controller goes completely unresponsive and has to be power cycled (this isn't). At least some of the logs I've seen indicate this is the actual problem.

[vladm]

@AlCalzone - Is this something that you could contact Zooz about, to investigate? Cause #2 is definitely a new phenomenon for the new firmware.

[AlCalzone]

It may be new for you, but it's not. I've added a detection and possible recovery for that state last september: #6244, and the issue it fixes is from early 2021.

FWIW, Zooz are aware, but it's not in their power to fix it.

[AlCalzone]

Does anyone except @vladm have driver logs showing the controller become unresponsive? Silabs are trying to figure out what could trigger this.

In the log above it might be lots of incoming traffic, but i wonder if that's the same for others with the same problem.

[andronat48]

@AlCalzone I wanted to create a nice log with a cloned ZST39, but it failed the upgrade from .19 to .21. I guess that's a separate issue but maybe you see something useful...

NVM cloned via simplicity studio, restart with a cloned stick around 2024-06-10T23:41:47.619Z. A few seconds in, the upgrade failed and the stick seems dead. COM port is visible, but not responding to anything and the blue light is off.

zwave-js-ui-store.zip

[kreucher]

I do.

Ran into this myself… and also could not downgrade back to 1.20 either via Z-Wave JS or Simplicity Studio. Overnighted another ZST39 stick and restored the NVM backup I (thankfully) made before doing the firmware upgrade. Back to a working setup.

So right now I have 2 sticks, one with 1.20 firmware that works, and one with 1.30 that shows the unresponsive controller loop issue.

Debug level logs attached for both, using the following steps:

Stopped HomeAssistant
Stopped Z-Wave JS

Unplugged ZST39 with 1.20 firmware
Plugged in ZST39 with 1.30 firmware

Started Z-Wave JS…
Waited for scan to complete…
“Scan completed” in UI

Sent various dim levels to a light (Node 51) – all worked

~08:36 PM PDT – started HomeAssistant

Toggled the same Node 51 on and off, dim level etc – all worked
Forced AdaptiveLighting off/on, sets many lights dim levels as once – worked

Started getting errors in Z-Wave JS log… controller unresponsive etc…

Toggled the same Node 51 on and off again – did not work
HomeAssistant errors “failed to call service light/turn/on unknown error”

HomeAssistant reporting status of ZST39 “Unresponsive” then “Ready” in a loop…

~08:38 PM PDT Borked…

Stopped HomeAssistant

Z-Wave JS log continues in a loop of resetting controller

Stopped Z-Wave JS

Unplugged ZST39 with 1.30 firmware
Plugged in ZST39 with 1.20 firmware

Repeated experiment steps above… worked fine with 1.20 (and still working).

Both 1.30 and 1.20 logs attached.

Let me know if anything else is needed, happy to help.
zst39-6874-debug.tar.gz

[rome1983]

I also have the endless unresponsive loop error.

zwave-js-ui-store.zip

[DijkstraSolutions]

Having done this just yesterday, I now concur with @AlCalzone. However, I noticed my RaspberryPi 3 is not having this issue where my 4s are having this problem.

root@jsserver231215:~# lscpu
Architecture:            aarch64
  CPU op-mode(s):        32-bit, 64-bit
  Byte Order:            Little Endian
CPU(s):                  4
  On-line CPU(s) list:   0-3
Vendor ID:               ARM
  Model name:            Cortex-A53
    Model:               4
    Thread(s) per core:  1
    Core(s) per cluster: 4
    Socket(s):           -

root@jsserver231210:~# lscpu
Architecture:            aarch64
  CPU op-mode(s):        32-bit, 64-bit
  Byte Order:            Little Endian
CPU(s):                  4
  On-line CPU(s) list:   0-3
Vendor ID:               ARM
  Model name:            Cortex-A72
    Model:               3
    Thread(s) per core:  1
    Core(s) per cluster: 4
    Socket(s):           -

Hope this use case helps someone either avoid, fix, or troubleshoot this problem. The OS is Ubuntu (Ubuntu 22.04.4 LTS) and Docker is CE (Docker version 26.1.4, build 5650f9b). Everything is up to date as of yesterday and I am using the zwavejs/zwavejs2mqtt:master image for my containers.

[AlCalzone]

I noticed my RaspberryPi 3 is not having this issue where my 4s are having this problem.

Please keep an eye on this. It would surprise me to learn that this is not purely a firmware issue.

[DijkstraSolutions]

I had to downgrade the 4's because they were just not functional, but I left the 3 updated. I will let you know if I see a failure. Race condition you think?

[droans]

I also had this error, but I was using a desktop PC, Ryzen 5 5600X, Gigabyte X570 Aorus Elite Wifi. My Z-Wave adapter was connected via a USB 3.0 hub connected to a USB 2.0 port.

[AlCalzone]

Race condition you think?

I'd say coincidence, but you never know.

[Indianb0y016]

Also began to encounter this issue after upgrading. I foolishly updated the firmware, when I never had issues at all on the previous firmware (1.20). A few days after, in the middle of the night, network became unresponsive (1.30). Update didnt seem to fail in zwavejs however it did show an error when finished. But no obvious problem with it. Im gonna try reinstalling the update in simplicity studio and will see what happens.

Update 1: Been approximately 24 hours after I flashed the new firmware using simplicity studio (windows) and I have NOT faced any timeouts. Prior to this, it would usually timeout in the night, though there was an instance where it timed out afternoon. I will post another update in maybe 72 hours if nothing happens or if it times out again.
Update 2: Well, it happened again. Around 10:45 this morning, it locked up again. So flashing via silicon labs software did not solve the problem. At this rate, Im going to assume this latest firmware update has exacerbated the controller lockup issues. Weirdly, I did not have these issues with 1.20. I will risk a downgrade now.

[Indianb0y016]

In proxmox, home assistant os as a vm, with everything up to date. USB passthrough is mapped correctly and I tested quite some time ago that the stick is still discovered when on different usb ports, which was the case.
The update did install, but zwavejs ui threw an error on completion but the stick still soft reset just fine and found all the nodes.
Behavior of the issues would happen randomly. As of now, I will have to wait about 24 hours to see if any other issues occur, having finished otw using simplicity studio.

[AlCalzone]

You're probably not using the latest version of Z-Wave JS. There was a fix for the timeout error despite successful upgrade.

[Indianb0y016]

True. The HA OS addon is usually a few versions behind, so it would be false for me to say "everything up to date." Regardless, it would appear that the firmware update introduced the bugs from 7.19.3 SDK, even though I never encountered them while on the 1.20 update (7.19.3 SDK). Very unlucky I suppose.

[ioguy]

Since switching to the ZST39 (7.19.3) from a 700 controller my network has been rock solid. Haven't seen any of the "unable to transmit bug".

I foolishly upgraded to 7.21.3 a couple days ago and also get the unresponsive loop. Only fix is to pull the USB stick (hard reboot). Downgraded by to 7.19.3 today, I'll wait until 7.22.X is released.

[ext4xfs]

FYI 7.22.0 is released

[mbarry1]

After reading these posts as well as the HA community posts it is clear that 7.21 introduced or exacerbated and issue with 800 series including the zst39.
If I understand correctly, zooz does not control the firmware even though they say you cannot load Silicon Labs glb firmware. I guess they do something small to it?? If that is the case how do these types of issues get reported and resolved by Silicon Labs. Is it through OEMs like zooz or some other mechanism? Is there something we can do to move this along?

[pshempel]

@AlCalzone

Unfortunately the tickets with Silicon Labs are all behind closed doors so to say. I can forward the information though if you post it here or send it to me.

114 Devices
26 sleeping/battery devices
6 Aeotec switches zwa038 or zwa037
1 Aetoec dimmer zen77
45 switches zen71 or zen76
1 Honeywell T6 Pro thermostat
29 motion/door sensors
6 locks in s0 mode
2 ring glass break
2 ring keypads
6 devices set with no security
Of the no security devices, one is an Aoetec power meter gen5
3 Ecolink chime+Siren
3 Aeotec indoor siren or doorbell
All but 1 device has all firmware updates done. The locks are Assure locks, unable to update firmware, because, Yale.
4 devices are Minoston power meter plugs, very little usage on these, usually are off

zwavejs_2024-06-17.log.gz

If there is any other information I could provide, please let me know. Also, I am willing to pay money to get any support possible. I can give access to the HA server if needed.

[DijkstraSolutions]

It took me a minute to get this compiled. I have 3 controllers for the single home, 191 has 46 devices, 192 has 55 devices and 193 has 45 devices.

Raspberry Pi 4s for 191 and 192, 193 is Raspberry Pi 3

Ubuntu 22.04.04, latest updates as of 6/14/2024
zwave-js-ui: 9.13.4.f07a854
zwave-js: 12.11.0

image: zwavejs/zwavejs2mqtt:master

Design is HA is running virtual on VMware [x86] (puke, right?) and I have 3 Pi's running Ubuntu with Docker and the HA controls all three. No networking issue, all enterprise equipment.

193 takes a while to start looping/not responding, 191 and 192 are so bad I had to back out the firmware.

193 does happen about once a day

All are running ping dead node (built in script) every 15 mins. It varies how often this is fired.

Z-Wave info only, easily 90% are not battery powered:

191:
Name                        Count
----                        -----
AEON Labs                      10
Aeotec Ltd.                     1
Eaton                          29
EcoNet Controls                 1
Sunricher                       1
Unknown manufacturer 0xXXXX     1	(Eaton Dimmer) [has issues sending, but receives fine]
Yale                            1
Zooz                            2

192:
Name                        Count
----                        -----
AEON Labs                       7
Aeotec Ltd.                     1
Eaton                          35
GE                              1
Minoston                        1
Sunricher                       3
Trane Corporation               1
Unknown manufacturer 0xXXXX     1	(Aeon light strip, old)
Yale                            3
Zooz                            2

193:
Name                        Count
----                        -----
AEON Labs                       2
Eaton                          30
EVA Logik                       1
GE                              2
HomeSeer Technologies           1
Unknown manufacturer 0xXXXX     5	(Eaton Dimmer) [has issues sending, but receives fine]
Yale                            1
Zooz                            1

It's a bit diverse and result of about 11 years of Z-Wave at this location.

[pshempel]

I just did an update to Z-wave JS UI version: 3.8.0. I seem to be much more stable now. I am not getting any more unresponsive controller errors, but still am getting tons of:

2024-06-18 16:04:02.052 INFO Z-WAVE: Setting user callbacks

At least the network is stable enough that the system can find all of the devices to be online before it starts getting user callbacks and things slow down from there.

[AlCalzone]

Setting user callbacks

That has nothing to do with the controller.

[pshempel]

zwave-js-ui-store.zip

These are the logs for yesterday. It now takes longer for it to go unresponsive and the only fix now to get anything back is to pull the device from the usb port and reinsert it to get it to become responsive

[YBonline]

I've been having the original jammed lockup issue as well. I was able to make it very rare by avoiding the use of my ZEN37. If you do a couple of hold events on a ZEN37, at least in my setup its basically guaranteed to cause a lockup. Unfortunately, my ZST39 is in a vacation rental, but I will be there tomorrow for a week to do some work on it, so I can try to get some logs if it will help. I tried firmware 1.30 last time I was there, but it was crashing every few minutes and went to 1.20 (I had no time to work on diagnosing any issue), combined with the removal of the ZEN37, has had relatively few issues.

[luckyjay]

What I've noticed on two setups I have is that FW: v1.30 SDK: v7.21.3 is much worse than FW: v1.20 SDK: v7.19.3. I've downgraded the firmware to 1.2 for stability and it seems to be okay like the OP mentioned.

Just started doing a test on another setup with firmware 1.3 and zwave js ui 9.14.0 (zwave js 12.11.0). Been up for a few hours so far, more than with previous versions of zwave js. However, one thing I noticed is that the status indicator in the top right of zwave js ui shows a "?" and "Unknown state", even though it appears to be working so far. That doesn't happen on firmware 1.2.

[AlCalzone]

I'd wait for a firmware based on 7.22.0 before trying the upgrade again. The workaround only helps with the failure to transmit, not the full lockup that's present in the 7.21.3 SDK.

[mbarry1]

@AlCalzone That makes sense. I hope 1. SI is actually working on this. 2. Zooz adopts it quickly. They took a while on 7.19.

[Botched1]

Historically Zooz has only done .1 releases, not .0.

So I would expect it o be 3-6 more months.

[AlCalzone]

I was told that they are planning to release a test build very soon, and follow it up with an official release if it works better than 7.21.3.

[luckyjay]

Another setup started having fits in the past hour. I'm going to downgrade everything and wait.

[AlCalzone]

I got a tip from Silicon Labs about enabling the hardware watchdog in the controller. This should help in the case where the controller locks up completely, because the watchdog should restart it. Should have an update ready tomorrow.

[vladm]

I’m not sure this gonna help, as when it locks, I tried disconnecting / connecting usb device in Linux and it won’t show up. Only physical disconnect helps.

[AlCalzone]

Guess we'll see 🤷🏻‍♂️

[frixo3190]

Hi, I just change the log verbose (so it restarts ZwaveJS) and .... my ZST39 became unresponsive ! I received the nodes reports, but can't send any order.
I've to unplug , wait 2min, and replug (if I don't wait the 2min, it doesn't work !).

Each time I change the log verbose (Info to None for instance) i've the trouble. I've tried 4 times !

But I have to confess that I just discover (As I write this message) a node that send report (W, kW, A, V) ..... every second ! ( the China Neo wall plug, that doesn't support the "0 sec" in config to disable it , there is a dropdown menu next to !)
So perhaps, my ZST39 became unresponsive because of that. I need to check again and give you log when I will have more time

[kfiducia]

I also have the unresponsive controller, adding my logs in case they help track down the problem.

zwave-js-ui: 9.14.1
zwave-js: 12.11.1

running on haos:
Core
2024.6.4
Supervisor
2024.06.0
Operating System
12.4
Frontend
20240610.1

34 zwave devices (mostly switches) including the controller ZST39 LR FW: v1.30 SDK: v7.21.3

i've had this controller lockup issue for weeks, i finally put the stick on a powered USB hub plugged into a wifi controlled outlet to "unplug" it when it goes unresponsive.

it happens most often when HA tries to trigger a scene, it seems to have something to do with a large volume of traffic, or perhaps it has to interact with a device it doesn't like. I have mostly Zooz 77 and 76s with some inovelli 2-1s, a ZAC38 (range extender), a ZEN37 scene controller, and a few others (can add the whole list if its useful). I am attaching a debug log from today where it recently locked up.

zwavejs_2024-06-24.log

I have many more log files I can post if they help. Will stay tuned to this thread in hopes there is some news.

Thanks to those of you who are helping to track this down, this is quite a frustrating puzzle.

[frixo3190]

The idea to plug the 800 controller on a Hub that can be disconnect/reconnect is sooo ingenius !

[kfiducia]

it pretty much resolved the lockup but occasionally the devices (lights) don't respond for 30 seconds while its rebooting, pair it with a notification so you know when its rebooting and its a pretty decent workaround.

fwiw - i bought an aoetec 5+ off ebay and restored my nvm to it, its been good since.

[AlCalzone]

Small update: Not sure how it came to this, but as of this morning, I am able to reproduce the situation where the controller hangs during transmission of a frame and becomes unresponsive afterwards. Working with Silabs right now to give them the information they need to reproduce and/or fix it.
Preliminary investigation on my part is that the controller starts beaming, even though the target node is an always-listening node, and does not stop until reset.

[darkbasic]

Man they need to start allowing firmware downgrades, too many regressions...

[AlCalzone]

is there a corresponding log message this behavior produces that we can look for?

Something with [SendDataAbort], then Controller missed Send Data callback. Attempting to recover, then a bit later The node is causing the controller to become unresponsive

[kfiducia]

I see the SendDataAbort in my logs but not every time the controller goes unresponsive, so I would agree, it may not be the only command causing the problems. That said, I do think based on the statistics in my logs (9 hits on the SendDataAbort for 19 The controller is unresponsive in the past week) I do think this would make a significant impact on the frequency of the lockup.

If it is useful, I see these messages in a slightly different order.

2024-06-20T03:27:54.803Z DRIVER » [REQ] [SendDataAbort]
2024-06-20T03:28:04.804Z CNTRLR   The controller is unresponsive
2024-06-20T03:28:04.807Z DRIVER   Controller missed Send Data callback. Attempting to recover...

Thanks for sending this so we can take a look at our own logs.

[AlCalzone]

To be clear, SendDataAbort is not the command causing the problem, it's part of what Z-Wave JS tries to prevent transmit attempts that are going on too long.

[kfiducia]

ah, my mistake, thank you for the clarification!

[AlCalzone]

Zooz shared a firmware based on SDK 7.22.0 for the ZST39, see link below. They tested upgrades and downgrades from and to 1.10, 1.20, and 1.30.
Still, be aware that this is a beta version (1.32), so proceed at your own risk. The stable release will be 1.40 after this has been tested a bit more.

ZST39_SDK_7.22.0_US-LR_V01R32_BETA.zip

[AlCalzone]

@YBonline if you have some driver logs on level debug, I'm happy to take a look what's going on there.

[pshempel]

I am still getting jammed notifications, but at least the controller right now is being restarted correctly and coming online. The issue when this happens is it drops messages that are out of sync.
The recovery is much better on the beta firmware. I no longer need to pull the controller from the USB to make it responsive. I have set debug level and logging now, will upload a log once I get a couple of jammed controller messages in the logs.

[alexruffell]

I upgraded to 7.22.0 beta and things started working only after a soft reset of the stick. I did the update using ZW PC Controller and it seemed way slower (4 minutes) than usual. Even the NVM backup took what seemed like an eternity (2 minutes) compared to previous attempts. Is the UART speed a variable here?

My controller was freezing up daily with 7.21.3.

[Matt1360]

Following up after two weeks, this has been a remarkable improvement. I can almost rely on automations working now, I use a lot of double tapping of light switches to turn them on for five minutes, and it has never worked as consistently as it is now with the beta FW.

It will still hang from time to time, but within ten seconds or so its soft reset has worked, and things are happy again. There's still room for improvement, but this has been incredible compared to the previous... well, quite a few months now.

Thank you for your work with Zooz on this!

[YBonline]

@AlCalzone Sorry it took me a little bit, I've been travelling and been a bit busy on other stuff. Still getting lots of "jammed", here's a sample on the latest beta firmware:
zwavejs_current.log

I also tried figuring if I could move my problemsome devices to LR but I've had no luck getting anything to pair with LR. SmartStart works fine if I don't use LR so I think I'm doing it right?

[jmwhite5]

I just tried the Zooz latest beta (ZST39_SDK_7.22.0_US-LR_V01R32_BETA.gbl) and after restoring my NVM, all I see is an infinite loop of controller unresponsive/ready messages (I did a Soft Controller Reset as was mentioned earlier. It did not help). The curious thing is that I had originally attempted to include 2 ZSE44s as Long Range, but had to revert when I went back to my Aeotec 700 stick. But for some reason, now those 2 sensors are showing up as Long Range with my Zooz 800LR stick. (I did restore the NVM, so shouldn't it just run as a normal inclusion? The provisioning entries are not set to Long Range. So not sure what's going on there.) In any case, this build is DOA for me 😔 I'm providing both driver and UI logs.

jmwhite5-logs.zip

Update: I had to spend an extra hour to figure out how to handle the 2 ZSE44s (+ a third one that was gen1 and doesn't support LR) sensors that now just were sitting in PROTOCOL_INFO state. I ended up EXLUDING all 3 and re-INCLUDING them to restore them. Once you know all these quirks, it's not too bad, but the fact that I even had any issues with those 3 ZSE44s in the first place is baffling. Reminder for anybody else trying this, when you exclude and re-include, you may want to refresh the browser since the zwave-js UI list may show you excluded devices and not show you newly included devices. Just refresh and you should get the correct list.

[pshempel]

@AlCalzone I had a complete lockup on the device yesterday. The only way to fix using the new beta version was to re-insert the device into the USB. I have included a few days of logs. There were intermittent lockups that seemed to be getting resolved by the automated reset, as you will see in the logs.

zwave-js-ui-store.zip

[duylong]

Too bad I didn't know I had to avoid updating to 1.30.

If I understand correctly, it's better to wait for 1.40, do you know the general delay? If it's several months, I'll go back to my 700 controller and wait, or if the downgrade works I could go back to version 1.20 but I'm afraid of bricking the key for nothing.

[AlCalzone]

@jmwhite5 to be honest, I did not understand what you were doing there and why you'd lose the interview progress for existing devices. Can you open a separate discussion and explain exactly what you did with which software in which order?

[duylong]

AFAIK that's just the default region setting. You may have to change region again after upgrading. Let me confirm with Zooz though.

I will wait, on their update site Zooz clearly indicates "Always download the files for your region only."

[AlCalzone]

Always download the files for your region only

This is very important for end devices, because once they are using the wrong frequency, there's no way to reach them anymore.
The stick will always communicate with the host via USB, even if it doesn't reach your end devices due to a wrong frequency.

According to Zooz, the difference between the US and the EU controller firmwares is the default region setting. Which you can easily change afterwards in Z-Wave JS UI.
FWIW, I'm using a US stick set to EU frequencies very regularly.

But I guess if you can wait, you might aswell wait to avoid any risk.

[jmwhite5]

@jmwhite5 to be honest, I did not understand what you were doing there and why you'd lose the interview progress for existing devices. Can you open a separate discussion and explain exactly what you did with which software in which order?

Sorry for the late response. I was traveling. In a nutshell, I was just trying the beta software on my Zooz 800LR stick. I normally run my aeotec 700 stick. I was hoping it would just take a few minutes to test, and here is the problem: My 3 800LR devices just went into a weird state (as I described earlier) when I was running the beta on my Zooz 800LR stick. So I decided to go back to my Aeotec 700 stick and my 3 800LR devices were still in a bad state and I had to exclude-include them to fix.

So I am motivated to test new software as it comes out, but if I keep having to fight those 3 800LR devices every time, it's going to be a pain. So my question is, how well have the 800LR devices been tested in this scenario? Are you aware of issues? Do you need more logs to troubleshoot?

[AlCalzone]

So far I'm not aware of any issues, but I also haven't tested an NVM migration with included LR nodes yet. I guess it would be best if you open a new discussion for investigation with some logs showing the behavior.

[AlCalzone]

I wanted to give an update on the 1.40 firmware, but I cannot find the thread right now. Zooz have the firmware ready, but aren't able to upload it to their website right now.
If anyone wants the firmware, please reach out to Zooz customer support.

[AlCalzone]

I am using Z-Wave JS UI

In that case, there is an option "allow bootloader only" in the settings. If you check that, Z-Wave JS UI starts in a mode that essentially only lets you re-attempt the firmware upgrade. It should let you recover a stick after an aborted or failed upgrade, just like the CLI tool does.

[mundschenk-at]

Had the same experience flashing from the ZWJS-UI, initial upgrade to final 1.40 failed, but the CLI tool worked.

[AlCalzone]

Any chance you have driver logs of the UI attempt? Both use the same methods internally, so this is unexpected.

[mundschenk-at]

I should have them, yes. I'll check.

[mundschenk-at]

@AlCalzone I've sent you the logs. I think the problem was that the driver tried to enable Smart Start midway through the update, and then the stick was in stuck in bootloader mode, so Z-Wave JS UI never came up to try again.

[mattleiphamellis]

Updated to 1.4.0 last night. Over 10 mains-powered Zooz devices that I've been unable to update over the air for months (persistent across soft resets, power cycles, excluding, etc.) were all able to update smoothly today, which I hope bodes well for others.

Some others asked which SDK version 1.4.0 was based on, which is addressed in the ZST39 1.4.0 release notes published today. The notes confirm that this release is based upon Z-Wave SDK 7.22.0.

[metril]

Updated to 1.4.0 and restored my configuration. The zooz 800 controller still locks up every few minutes. It does seem to recover if you wait, but the lockups keep happening. The problem is not resolved.

PS. And now the controller sometimes goes into a "controller is not responsive" and "controller is ready" loop.

[m3ki]

I was wondering if there is any relation to this issue zwave-js/zwave-js-ui#3639
I was installing Zooz 800 LR on rpi 2 b and zwave js would lock up on me when I would save the configuration.
I I were to downgrade to linux kernel 6.1 lock-ups are gone.

[AlCalzone]

I don't think so. This issue is about the firmware of the stick locking up, not the host.

[AlCalzone]

Zooz told me they have a beta version 1.41 available, based on SDK 7.22.1, which should improve stability of the 800 series even more.
If anyone wants to beta test, please reach out to their support.

[Tschuuuls]

And they seem to have dropped support for 700 series 7.22 apparently? #6512 (comment) EDIT: Nevermind.

[AlCalzone]

No, just split 700 and 800 series into two different SDKs.

[marckessels]

the beta 1.41 is really an improvement! After one day, no more soft resets were needed, compared to the previous versions where i needed to do it multiple times per day.

Interviewing nodes still fails once or twice, but then it works, whereas it was impossible with previous versions. So i think the problem has not been solved completely, but it now seems to have reached an acceptable level.

[luckyjay]

I see that Zooz released 1.50 at https://www.support.getzooz.com/kb/article/1158-zooz-ota-firmware-files/, and interestingly removed all older versions.

[mbarry1]

Just updated via simplicity studio and no issues so far. Hopefully they are moving in the right direction.

[vladm]

just tried to update via homeassistant, it interrupted fw upload process pretty soon after start, and I think I bricked my stick. in the logs of zwave-js:

2024-09-12 16:39:34.375 INFO Z-WAVE: Controller status: Driver: Failed to recover from bootloader. Please flash a new firmware to continue... (ZW0100)
2024-09-12 16:39:34.376 INFO Z-WAVE: Restarting client in 1 seconds, retry 1
2024-09-12 16:39:35.379 INFO Z-WAVE-SERVER: Server closed

[AlCalzone]

@YBonline you may still have too much traffic. Look into https://zwave-js.github.io/node-zwave-js/#/troubleshooting/first-steps
Jammed = unable to transmit is not the same as unresponsive = not responding to Z-Wave JS.

[duylong]

No problem on my side with Zwave JS UI, sorry for you.

[vladm]

@vladm Try Z-Wave JS UI, enable "bootloader only" mode in settings.

ty for that, I was able to flash FW using zwave-js. Interesting observation, after I put it in bootloader-only mode and selected file to flash, it brought up progress dialog, which is just stuck at 0%.

I checked docker logs and saw that it actually flashed it successfully and started network. opened another tab with zw-ui and it shows all good. The original windows with 0% dialog is still hanging though :)

2024-09-13 10:39:47.363 INFO Z-WAVE: Controller status: Bootloader is READY
2024-09-13 10:39:47.363 INFO Z-WAVE: Bootloader is READY
2024-09-13 10:41:07.793 DEBUG SOCKET: Event ZWAVE_API emitted to 673iVksp_qJMPH4yAAAB
2024-09-13 10:41:07.796 INFO Z-WAVE: Calling api firmwareUpdateOTW with args: [
  {
    name: 'ZST39_SDK_7.22.1_US-LR_V01R50.gbl',
    data: <Buffer eb 17 a6 03 08 00 00 00 00 00 00 03 01 01 00 00 fa 06 06 fa 10 00 00 00 6e 2e 02 00 d9 70 f3 d5 f0 a6 40 99 0b 75 4e 43 f9 07 07 f9 24 00 00 00 40 d8 ... 143058 more bytes>
  },
  [length]: 1
]
2024-09-13 10:41:29.817 INFO Z-WAVE: Success zwave api call firmwareUpdateOTW { success: true, status: 255 }
2024-09-13 10:41:29.821 INFO Z-WAVE: Controller status: Driver: Restarting driver after OTW update... (ZW0100)
2024-09-13 10:41:29.821 INFO Z-WAVE: Restarting client in 1 seconds, retry 1
2024-09-13 10:41:30.822 INFO Z-WAVE-SERVER: Server closed
2024-09-13 10:41:30.823 INFO Z-WAVE: Client closed
2024-09-13 10:41:30.830 INFO Z-WAVE: Connecting to /dev/ttyACM0
2024-09-13 10:41:30.830 INFO Z-WAVE: Setting user callbacks
Logging to file:
        /usr/src/app/store/logs/zwavejs_2024-09-13.log
2024-09-13 10:41:33.310 INFO GATEWAY: Driver is READY
2024-09-13 10:41:33.311 INFO Z-WAVE: Z-Wave driver is ready
2024-09-13 10:41:33.311 INFO Z-WAVE: Controller status: Driver ready
2024-09-13 10:41:33.316 DEBUG Z-WAVE: [Node 001] Binding to node events
2024-09-13 10:41:33.331 DEBUG Z-WAVE: [Node 001] Has been added to nodes array
2024-09-13 10:41:33.332 DEBUG Z-WAVE: [Node 002] Binding to node events

[zhazell]

Updated mine to the 1.5 firmware. Received an email yesterday from Zooz announcing the release. Did the update from Z-Wave JS UI and it seemed to go quick and easy. However it did get "stuck" on waiting for controller to reboot at like 99.4%. Refresh of the page was the only way to clear the pop up message and all devices are communicating.

[mundschenk-at]

Had the same experience as @vladm. Stick initially got stuck in bootloader mode again (I'll check the logs why later - Edit; oh, I didn't realize #7076 was still open), flashing in "bootloader only" mode worked fine, but didn't update the UI.

[andronat48]

1.50 has been very smooth for a couple of weeks.

[mbarry1]

very smooth for me also with 70+ devices

[Botched1]

I still see on average 8-10 jammed events a week on the 1.50 firmware, but nothing I can't live with.

<style type="text/css"></style>

entity_id	state	last_changed
sensor.zwave_controller_status	jammed	2024-09-26T23:17:59.630Z
sensor.zwave_controller_status	jammed	2024-09-27T23:45:27.511Z
sensor.zwave_controller_status	jammed	2024-09-29T16:02:14.778Z
sensor.zwave_controller_status	jammed	2024-09-29T22:59:17.954Z
sensor.zwave_controller_status	jammed	2024-09-30T00:29:32.054Z
sensor.zwave_controller_status	jammed	2024-09-30T00:51:04.214Z
sensor.zwave_controller_status	jammed	2024-09-30T01:07:46.824Z
sensor.zwave_controller_status	jammed	2024-10-03T22:48:34.962Z

[m3ki]

What's the consensus on the 1.5fw? Is it stable? Any lockups or jams?

[Matt1360]

It has been much more stable than previous versions. I wouldn't say it's bulletproof, however. The controller seems to reset when it needs to, which doesn't smell like the underlying problem is fixed, but it does recover. Sometimes it might take a bit (~30 sec?).

I use an automation on most of my switches that let me double tap them to turn them on for five minutes. It will first set the LED colour on the switch, and then turn the light on. Most of the time it works now, but it's definitely not perfect. Sometimes it won't take, and other times it will... eventually.

Coming from previous versions where this was literally unusable, this is a huge improvement. I'm hoping it will continue to improve over time.