Zigbee devices going offline with zigbee-addon 3.2.0

[nailyk-fr]

Zigbee devices going offline with zigbee-addon 3.2.0

[cdjackson]

Firtstly, I would suggest that firmware 6.0 will not work with most new devices - so I'm surprised this really works at all. You need FW 6.4 or newer to work with most ZB3 devices.

However, that's not probably the issue here - there seems to be a memory problem in the NCP as I can see it is out of buffers (status=EMBER_NO_BUFFERS). I'm not really sure what will cause this - maybe you have set some of the configuration to use very large network which will consume memory in the NCP and I'd suggest to check your configuration and reduce the network size.

None of this will be related to the binding though, so if you use the 3.1 settings it should be fine (it hasn't changed for a long long time).

[cdjackson]

The number of devices joined to the network will have no impact on memory use in the coordinator so removing devices will likely not help. Memory is allocated based on the coordinator configuration and not when devices are added to the network.

I would strongly suggest to reduce the network size from 100 to something smaller - this will allocate a lot of memory, and I suspect that your device doesn't have the capability to support this.

[nailyk-fr]

Thanks a lot for your fast answer !

Indeed, the controller is setup on 'High Ram'. Here is the controller configuration (I carefully reported all the configuration between the old setup and the new one).

UID: zigbee:coordinator_ember:6fbabd96fd
label: Coordinator - Ember Coordinator - Elelabs
thingTypeUID: zigbee:coordinator_ember
configuration:
  zigbee_port: /dev/serial/by-id/usb-1a86_USB2.0-Serial-if00-port0
  zigbee_channel: 25
  zigbee_initialise: false
  zigbee_concentrator: 1
  zigbee_trustcentremode: TC_JOIN_SECURE
  zigbee_extendedpanid: 8B73F7F5B45C722D
  zigbee_baud: 115200
  zigbee_flowcontrol: 1
  zigbee_panid: 35854
  zigbee_powermode: 1
  zigbee_txpower: 8
  zigbee_networksize: 100
  zigbee_linkkey: 5A6967426565416C6C69616E63653039
  zigbee_childtimeout: 172800
  zigbee_networkkey: B401068DAA25F9A54EE0F81CEE951E20
  zigbee_meshupdateperiod: 86400

For testing purposes, I can try to remove some devices but it worked perfectly for some days. Though, on my 3.1.0 setup, it was much less devices.

[cdjackson]

I guess based on your comments elsewhere that this is resolved?

As noted, this issue is documented in the README - it states that if you set this number too high, it will impact on the number of packet buffers which is exactly the error in your logs. The error in your logs is status=EMBER_NO_BUFFERS.

[nailyk-fr]

Indeed !
So far, everything is working properly.

However, I did basic debugging mistake... I changed the network size & upgraded the key at the same time. So I am now unsure what was the real fix.

My guests went out today so I can break the system again.
Next step is to increase network size again and wait couple of weeks if the problem will come back.

Maybe the README can be adapted this way ?
To add in the zigbee_networksize section this it can cause status=EMBER_NO_BUFFERS error if set too big ?

[cdjackson]

However, I did basic debugging mistake... I changed the network size & upgraded the key at the same time. So I am now unsure what was the real fix.

For sure - your problem was the network size. When you increase the network size, the binding will make the tables larger, this in term uses more memory, and in turn this causes the error that we saw in your logs. Changing the network key will not change memory allocation.

Maybe the README can be adapted this way ?

Yep - I'm happy with adding something along these lines. I assume you are offering to create a PR :) Thanks.

Do you have any logs from your binding startup during this error? If so can you provide them - I want to see if it's possible to detect this problem and raise a more useful message during the startup. This should be possible, but I want to see if in your situation I see the responses that I expect in this case.

[nailyk-fr]

For sure - your problem was the network size. When you increase the network size, the binding will make the tables larger, this in term uses more memory, and in turn this causes the error that we saw in your logs. Changing the network key will not change memory allocation.

Sorry, I meant I upgraded the firmware of the key and change the network size to 25 at the same time.

I do not store the logs. As it is quite verbose, they are on a tmpfs...
Tho, if the network key increase produce the error again, I will surely provide a log !

[cdjackson]

Sorry, I meant I upgraded the firmware of the key and change the network size to 25 at the same time.

Ah - ok. I still think that the fix will be the network size. As far as I know the firmware created by ELE labs doesn't use all the available memory, although this might have been fixed in more recent versions.

[cdjackson]

I believe this issue is resolved as per the discussion above.

[nailyk-fr]

Hello @cdjackson

Sorry for super late answer, I was thinking about updating today 😉
For some weeks (month?) now, I switched back the Network size setting to 'Enormous' in order to reproduce the issue :

And, so far, no issue at all. Once, after some people pushed a lot the buttons on the Philips remote, we experienced a huge delay. But, I believe, it was because the service was still processing all the button push.

So, it seems to me, the main issue was the firmware upgrade of the key. Is there any way to downgrade the firmware to prove this theory ?

[cdjackson]

I don't know what firmware you are now using, but I'm very confident that your original issue was the network size - this was very clear from the logs you provided.

New firmware may also have allocated more memory, so this may also now allow you to use a larger network, but I really can't say without knowing exactly how the firmware is compiled.