[EMBER] Frequent drop offs of devices

[zsmartsystems]

[EMBER] Frequent drop offs of devices

[cdjackson]

Ok, thanks. I think we can conclude that it’s probably no worse than it has been and not that there isn’t an issue. I pinged Silabs earlier today so hopefully we can improve this.

…

On 7 Aug 2018, at 18:37, Scott Rushworth ***@***.***> wrote: I can't get any more logs because the bulbs have been behaving for about 2 days now! I fell asleep while testing the script, and Zigbee was offline. The bulbs all panicked and turned on. When I started Zigbee back up, they came online and haven't dropped. I am running the latest version, so either something was fixed (doubtful) or the mesh rebuilt itself and the bulbs are happy again. I had thought of doing this intentionally, but never got around to it! I think this can be closed, unless you think there is something more to it. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#346 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA_kQ1xHob-jPRoS73r4clXGky-iXEeEks5uOdBegaJpZM4VsDQS>.

[cdjackson]

I don't think there have been any changes in this area between 13 and 14, so I think this is the same issue. I have a ticket open with Silabs - their documentation is very poor and doesn't describe the rejoin policies (their support people also can't answer the question and they need to check with the source code!).

[5iver]

I don't think there have been any changes in this area between 13 and 14

OK... the drop offs were definitely happening in both, and previous, but seemed to be more frequent in more recent versions. Using the current snapshot now (2.4.0.201808020834 w/14), my bulbs are in a continuous cycle of drop off and re-pairing at least a couple times a day. Was there a commit that caused this... maybe SPI? I'm asking so that I can rollback to a stable version until you get a fix in.

[cdjackson]

As I said, I don’t think there have been any changes to the Ember handler between 13 and 14 that would cause this. In fact there have been very few changes between 13 and 14. I don’t think you are using SPI, so even if there have been changes to SPI it would not affect you.

[cdjackson]

In the log you attached, can you tell me what time a device left and rejoined. From a quick scan I don't see anything in the log so it would help to have a time to look at if possible...

[5iver]

In the log you attached, can you tell me what time a device left and rejoined.

No, but I think I can reproduce it. Power cycling a bulb will occasionally cause a drop off. Frustrating when trying to get a cluster of 4 on one switch to all get back on the network! One will join and another will drop. Otherwise, I can just turn on logging and wait for a light to turn on (they do that when they drop).

As I said, I don’t think there have been any changes to the Ember handler between 13 and 14 that would cause this

I was meaning further back than 13. I just put in 12 to see if there are less drop offs.

[cdjackson]

You will not be able to go back further or the binding will not start. In any case, as I said, I believe that this issue is related to the rejoin policy. This has NEVER changed. Silabs documentation is very (VERY!) poor and their tech support have not yet been able to answer my question on the policies. Maybe once this is worked out we can look at it. The log would be interesting just to confirm this. I don’t expect there to be anything much logged, which again shows that it’s not directly the library that’s doing something wrong, but probably incorrect policies which are not documented.

…

On 5 Aug 2018, at 15:51, Scott Rushworth ***@***.***> wrote: In the log you attached, can you tell me what time a device left and rejoined. No, but I think I can reproduce it. Power cycling a bulb usually causes a drop off. Otherwise, I can just turn on logging and wait for a light to turn on (they do that when they drop). As I said, I don’t think there have been any changes to the Ember handler between 13 and 14 that would cause this I was meaning further back than 13. I just put in 12 to see if there are less drop offs. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#346 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA_kQ1X2uS_zv6EhZnq0cGK6FJWhjlWsks5uNwZhgaJpZM4VsDQS>.

[5iver]

In any case, as I said, I believe that this issue is related to the rejoin policy. This has NEVER changed.

I hear you... but in my experience, it got a lot worse recently, just not sure exactly when. Maybe there's just some new RF interference I haven't found. Hopefully Silabs gets back to you soon! I'll get some logs together. Which reminds me, I got that upgrade script together. This will give me another chance to test it.

[cdjackson]

I know it’s easy to think that it’s something in the binding that must have changed when things change, but it’s not always the case ;) I’ll give Silabs a few more days and then send another ping. They agreed with me that it sounds like rejoin issues, so if we can work out what the configuration is meant to look like, hopefully it can be sorted reasonably easily… The script would be really cool - thanks :)

…

On 5 Aug 2018, at 16:09, Scott Rushworth ***@***.***> wrote: In any case, as I said, I believe that this issue is related to the rejoin policy. This has NEVER changed. I hear you... but in my experience, it got a lot worse recently, just not sure exactly when. Maybe there's just some new RF interference I haven't found. Hopefully Silabs gets back to you soon! I'll get some logs together. Which reminds me, I got that upgrade script together. This will give me another chance to test it. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#346 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA_kQwschF8Rlt5XuHEyDU137jJ5b4UDks5uNwq3gaJpZM4VsDQS>.

[5iver]

I can't get any more logs because the bulbs have been behaving for about 2 days now! I fell asleep while testing the script, and Zigbee was offline. The bulbs all panicked and turned on. When I started Zigbee back up, they came online and haven't dropped. I am running the latest version, so either something was fixed (doubtful) or the mesh rebuilt itself and the bulbs are happy again. I had thought of doing this intentionally, but never got around to it! I think this can be closed, unless you think there is something more to it.

[5iver]

The stable network only lasted a couple days. I just happened to catch a bulb that dropped, rediscovered it, turned it off, enabled logging, then within 15 minutes it popped on again. If there's anything in the log, it should be in here! Device is 7CE52400000B7EED, 16952.
081418 zigbee drop.zip

[cdjackson]

Nothing obvious in here. At 15:24:01 the device doesn’t seem to respond, although it’s ok at 15:22:11. There doesn’t appear to be anything happening or logged. It does seem to recover, so this might just be a one off. It also sends a bunch of announce messages at 15:25:55 which might be when you powered it off/on or something (when a device is powered on it will normally send an announce message). Nothing wrong that I can spot though...

…

On 14 Aug 2018, at 20:41, Scott Rushworth ***@***.***> wrote: The stable network only lasted a couple days. I just happened to catch a bulb that dropped, rediscovered it, turned it off, enabled logging, then within 15 minutes it popped on again. If there's anything in the log, it should be in here! Device is 7CE52400000B7EED, 16952. 081418 zigbee drop.zip <https://github.com/zsmartsystems/com.zsmartsystems.zigbee/files/2288239/081418.zigbee.drop.zip> — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#346 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA_kQxhzLfG3xpCx7Ihb0W93OGQ5RJH2ks5uQyfTgaJpZM4VsDQS>.

[5iver]

I never power cycled it. It turned itself on when it dropped off the network. After dropping, sometimes they will do single flashes, which I have never seen before. When they do this, I've had good success kicking off a discovery and having a dropped bulb come back online and controllable. I did this, turned it off, then started logging. It then fell off the network again while logging. As sometimes happens, this bulb then came back online by itself, without manual discovery. This is probably what you are seeing at 15:25:55, when it came back online. I'm not sure if the bulbs are turning on when they drop or come back online.

The only time I am used to seeing flashing, is when the bulbs are first paired, and they flash 3 times slowly. The single flashes are really quick. Sometimes a few within a minute. Never seen this with an older version of the binding.

I kind of expected there wouldn't be anything in the log, but this one definitely had a bulb dropping in it. It would have been a minute or two from the end, so 15:24 sounds about right.

[cdjackson]

The announce message can be sent for other reasons than a power cycle. In general, the library doesn't do too much if it receives an announce for a device it already knows - it should make sure that the network address hasn't changed, but that's about it. I couldn't see anything wrong - and given that at least in this log the device does start responding again later, I don't think there's a problem with the software...

I've had one of those weeks and not sorted out the package to send you yet - I'll try and get this done in the coming days...

[AfromD]

on change of WandschalterSchlafzimmer_Deckenlicht_SwitchOnoff (001fee00000016be), the items
HUELichtleisteSchrank_Color (0017880101210d6a)
HUESchlafzimmerStehlampe_Color (0017880102548f94)
SchranklichtSchlafzimmer_SwitchOnOff (84182600000c8aa7)
NachttischRechts_Color (00158d00011e871a)
NachttischLinks_Color (00158d0000af412c)
Are supposed to be switched on or off by a rule

I have attached a log, where only NachttischLinks_Color works as it should. All the others look normal in OH, but do not react in real life.
emberlog.txt

[cdjackson]

Please can you say what version of the library you are using. If it is not at lease 1.1.8 then please use this. I would also like to know what settings you use. If you have a sniffer log, then please provide this, and also the XML file from the network so I can understand what devices you have (I don't really understand what you have written above - sorry). It may also be useful to provide the actual device types as well.

…

Sent from my iPhone

On 14 Jan 2019, at 16:00, AfromD ***@***.***> wrote: on change of WandschalterSchlafzimmer_Deckenlicht_SwitchOnoff (001fee00000016be), the items HUELichtleisteSchrank_Color (0017880101210d6a) HUESchlafzimmerStehlampe_Color (0017880102548f94) SchranklichtSchlafzimmer_SwitchOnOff (84182600000c8aa7) NachttischRechts_Color (00158d00011e871a) NachttischLinks_Color (00158d0000af412c) Are supposed to be switched on or off by a rule I have attached a log, where only NachttischLinks_Color works as it should. All the others look normal in OH, but do not react in real life. emberlog.txt — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[cdjackson]

Unfortunately this log is some sort of openHAB log (I'm not super sure how it is generated) and it is probably from quite an old version of the library as debug messages are not in the format to be processed.

I will await further information before trying to work out what this all means.

[AfromD]

This was from the events log in the web front end. The library is the current 1.1.8. I am not sure how to get a sniffer log, but I uploaded the normal log and the xml.

This occurence was different from ones with the older library. The devices did not come back after several restarts of OH. They came back after I reassociated them with the network (reset the Hue lights with the dimmer remote whilst having OH discover new zigbee devices). They did not show up in the inbox, but started working again straight away. It seems OH was never aware of them gone missing. The debug log only shows them not reacting to commands. I stopped debug logging before I restarted OH.

My settings:
Bridge zigbee:coordinator_ember:f737c1d4 "Ember EM35x Coordinator" [
zigbee_port= "/dev/ttyAMA0",
zigbee_baud= 57600,
zigbee_flowcontrol= 2,
zigbee_powermode= 1,
zigbee_txpower= 8,
zigbee_channel= 25,
zigbee_initialise= false,
zigbee_panid= 2232,
zigbee_macaddress= "000D6F000EE3DC01",
zigbee_extendedpanid= "824AB3EDD3867E08",
left out the network key, but it is defined.

zigbee-network--zigbee_coordinator_ember_f737c1d4.xml.txt
openhab.log

[cdjackson]

This was from the events log in the web front end.

It's a strange format - it would be nice to just have a standard log...

I am not sure how to get a sniffer log

You would need a sniffer :) Really without this it will likely be difficult to know what is happening - if they are leaving the network, then the logs generally won't show what is really happening.

My settings

What do you have the child timeout configuration set to? This is quite important.

[AfromD]

log: I attached a standard openhab.log from the logs directory to the post above.
sniffer: I do have a TI dongle from china with a programmer, which firmware do I need to flash on it?
settings: You are right, the parameter has not made it into my thing definition yet, so I guess I am using the default, Paper UI says 1 day.

[cdjackson]

I do have a TI dongle from china with a programmer, which firmware do I need to flash on it?

Sorry - I'm not really familiar with this - I use the Ember tools.

settings: You are right, the parameter has not made it into my thing definition yet, so I guess I am using the default, Paper UI says 1 day.

The default in the Ember NCP is 5 minutes if you don't set it. PaperUI may say 1 day as that's how I prefer it to be configured, but if you are using text definitions, then I think it's likely 5 minutes and I suggest that you change this.

[AfromD]

I also have a spare HUSBZB-1 dongle.

I have put zigbee_childtimeout= 86400 in my thing definition now.

[cdjackson]

I also have a spare HUSBZB-1 dongle.

Cool - this I can help you with.

See this repository for documentation and software -:

https://github.com/zsmartsystems/com.zsmartsystems.zigbee.sniffer

[cdjackson]

@openhab-5iver since you opened this, can you comment if this is still an issue? I think there have been a lot of changes in the past 6+ months, so I would prefer to close this if you agree? If not, let's get a status of the current situation...

[5iver]

I replaced 6 bulbs with zwave dimmers and switches. 5 of them were outside on the patio, but they were within 5ft of bulbs inside the house. I don't know for sure, but it's possible that none of them had a direct route to the coordinator. There was an update around the same time, so I don't know if removing these bulbs was a factor or not, but there were still some issues after the update. I still have one bulb outside, but it's been turned off for a few months, so I just turned it on and it paired up, so I'll see how it behaves (have a zwave dimmer on a shelf 😄). It is the furthest away from the coordinator and has 3 bulbs and an outlet about 25ft away.

However, the current situation is MUCH improved with the updates! I've had only 3 bulbs become unresponsive in the last month. At the worst, none of the 25 bulbs (at the time) would last more than a day. You've made significant progress... thank you!

[cdjackson]

Thanks Scott - I'm sure you'll let me know if there are further problems 😄, but it's good to hear things are pretty stable at the moment.