TI Z-Stack Home 1.2.2 (Z-Stack Core 2.6.3) Support

[tlaukkan]

TI Z-Stack Home 1.2.2 (Z-Stack Core 2.6.3) Support

[cdjackson]

@chris-hatton is this the firmware you're working with, or is this a different version again?

[tlaukkan]

I think it is the same Z-Stack but @chris-hatton was using Z-Stack Home where as @presslab-us was testing Z-Stack Linux Gateway. I need to take a look at what Z-Stack Linux Gateway is exactly. @presslab-us you can try to use the Z-Stack Home if it is intended for same hardware. I got impression from Chris that it works.

[tlaukkan]

It seems that the TI Z-Stack Linux Gateway is TI middleware implementation of the IP Network -> ZigBee Network Gateway specification. Understanding ZigBee Gateway:
https://docs.zigbee.org/zigbee-docs/dcn/09-5465.pdf

TI Z-Stack Linux Gateway enables one to transform combination of ubuntu box and CC253x ZNP to such gateway: http://e2e.ti.com/support/wireless_connectivity/f/158/t/347669

I think this quite nice concept but I am a bit uneasy about attack surface increase in general. Is it not much safer to have serial port interface to ZigBee network than accessing ZigBee Network over IP Gateway. Attacker gaining access to any device in that IP network can try to hack the IP gateway where as serial port access requires hacking the specific Home Automation server.

The really good thing is that ZigBee Gateway has standardized RPC interfaces for ZigBee network operations which so far has not been the case. Hooking zigbee4java to these RPC interfaces is well worth the effort as soon as we can find some nice off the shelf ZigBee gateway boxes you can just plug to your wired network.

[presslab-us]

Hi Tommi, I meant not to use the TI Gateway itself, but instead to just use the HEX file included in the download. This way it is easy to program the USB CC2531 device and it's not necessary to use IAR for compiling the ZNP firmware. Also using TI's firmware alleviates any concern of distributing a copyright HEX file.

I think supporting the RPC interface of the TI Gateway would be nice too, but it's not what I meant on this issue. Sorry for not being clearer.

Ryan

[chris-hatton]

Hi all,

Ryan; I may have missed you off a previous email regarding Z-Stack Home / Gateway firmware.
I think we are all agreed that targeting an updated (and more accessible) dongle firmware is desirable.
Please take a look below: I think you’ll agree that the ‘Linux Gateway’ firmware doesn’t look like the way to go...

Available CC2531 Firmwares

There are several downloadable Z-Stack packages made available by Ti (at http://www.ti.com/tool/z-stack), these are:

Z-STACK-LINUX-GATEWAY,
Z-STACK-HOME,
Z-STACK-LIGHTING,
Z-STACK-MESH,
Z-STACK-ENERGY

I downloaded all of these modules side-by-sideand used the following BSD 'Tree' command line to produce an informative listing of all firmware files contained within:

tree -h -P ".hex|CC253.bin" --prune > Z-Stack_CC253x_Firmwares.txt

Gives:

├── [ 272] Z-Stack\ Gateway
│ └── [ 170] misc
│ └── [234K] CC2531-GW-ZNP_38724.bin
├── [ 544] Z-Stack\ Home\ 1.2.1
│ └── [ 170] Projects
│ └── [ 374] zstack
│ ├── [ 272] ZAP
│ │ └── [ 306] ZNP-HexFiles
│ │ ├── [681K] CC2530ZNP-Pro-Secure_LinkKeyJoin.hex
│ │ ├── [681K] CC2530ZNP-Pro-Secure_Standard.hex
│ │ ├── [681K] CC2530ZNP-Pro.hex
│ │ ├── [681K] CC2531ZNP-Pro-Secure_LinkKeyJoin.hex
│ │ ├── [681K] CC2531ZNP-Pro-Secure_Standard.hex
│ │ └── [681K] CC2531ZNP-Pro.hex
│ └── [ 170] ZNP
│ └── [ 340] CC253x
│ └── [ 136] bin
│ ├── [ 23K] CC2530ZNP-SB.hex
│ └── [ 23K] CC2531SB.hex
├── [ 442] Z-Stack\ Lighting\ 1.0.2
│ └── [ 102] Projects
│ └── [ 238] zstack
│ └── [ 136] ZLL
│ └── [ 136] SampleApp
│ └── [ 476] CC253x
│ └── [ 170] hex
│ ├── [601K] SampleBridge-Dongle.hex
│ ├── [576K] ZllLight-ZLIGHT.hex
│ └── [459K] ZllRemote-ZLLRC.hex
└── [ 544] Z-Stack\ Mesh\ 1.0.0
└── [ 102] Projects
└── [ 306] zstack
├── [ 204] ZAP
│ └── [ 170] ZNP-HexFiles
│ ├── [ 204] CC2530
│ │ ├── [681K] CC2530ZNP-Pro-Secure_LinkKeyJoin.hex
│ │ ├── [681K] CC2530ZNP-Pro-Secure_Standard.hex
│ │ └── [681K] CC2530ZNP-Pro.hex
│ ├── [ 204] CC2531
│ │ ├── [681K] CC2531ZNP-Pro-Secure_LinkKeyJoin.hex
│ │ ├── [681K] CC2531ZNP-Pro-Secure_Standard.hex
│ │ └── [681K] CC2531ZNP-Pro.hex
│ └── [ 306] CC2538
│ ├── [681K] CC2538ZNP-SPI-Pro-Secure_LinkKeyJoin.hex
│ ├── [681K] CC2538ZNP-SPI-Pro-Secure_Standard.hex
│ ├── [681K] CC2538ZNP-SPI-Pro.hex
│ ├── [681K] CC2538ZNP-UART-Pro-Secure_LinkKeyJoin.hex
│ ├── [681K] CC2538ZNP-UART-Pro-Secure_Standard.hex
│ └── [681K] CC2538ZNP-UART-Pro.hex
└── [ 170] ZNP
└── [ 340] CC253x
└── [ 136] bin
├── [ 23K] CC2530ZNP-SB.hex
└── [ 23K] CC2531SB.hex

28 directories, 26 files

A comment on each module:

Z-STACK-LINUX-GATEWAY
Provides a ZigBee stack comprised of a Linux Application alonside a much smaller firmware file for the CC2531.
The documentation doesn't make explicit what layers reside in the CC2531 firmware and which in the Linux Application, but it can be readily seen that the .hex file for the CC2531 is much smaller than the one included in Z-Stack Home & Mesh. This, along with the lack of secure variants would suggest that much of the functionality is left implemented by the Linux portion. The source code is unavailable for the Linux portion and in my experiment with running the provided Linux binary on a couple of Embedded systems I have here, it turns out the Gateway is compiled specifically for Ti’s ‘Beagle-Bone’ board (IIRC ARM v5 w/ soft-float ABI). The gateway is a nice concept but given the above it’s usefulness appears limited for us?

Z-Stack Linux Gateway Release Notice
http://e2e.ti.com/support/wireless_connectivity/f/158/t/347669

Z-STACK-HOME & Z-STACK-MESH
These packages both contain the same firmware like the firmware, which they tellingly refer to in their documentation as plain ‘Z-Stack’ or ‘Z-Stack Core’. This is almost certainly what we outght to be targeting. Ti's description is:
"Z-Stack Home is TI's ZigBee Home Automation (ZHA) compliant protocol stack for the CC2630, CC2530 and CC2538 System-on-Chip. Z-Stack Home notables include: Based on Z-Stack, ZigBee PRO 2012 protocol stack with ZCP (ZigBee Compliant Platform) certification and 802.15.4 MAC certification.”.
It is larger than ‘Gateway’ at 681k and comes with two variants at differing levels of security.
The reason why the firmware is provided twice in two packages seems to just be target audience: The documentation that comes in the Home and Mesh packages caters for the different use cases of Home Automation and plain Mesh Networking respectively.

Z-STACK-LIGHTING
Z-STACK-ENERGY
The purpose of these packages appears to be just to provide documentation and small application software examples of how to drive ZigBee compliant lighting and renewable energy devices.

I would encourage you to download these and look for yourselves as well but it seems conclusive that we ought to be targeting CC2538ZNP-SPI-Pro.hex and its secure variants.

I have already run ZigBee Console on the new firmware -and it does communicate - but I didn’t have any ZigBee target devices available at that time to complete a full test. Will do soon.

Must dash now,
Thanks,
Chris

[presslab-us]

@chris-hatton Very informative, thanks! I see the Z-Stack Home is dated 23-FEB-2015 and I have not seen this version yet. This should be a newer version yet than the Gateway one.

I don't see the Gateway firmware in HEX in your list, although I do see the BIN. The HEX file will be larger than the BIN file as they are different file formats. The BIN is meant to be used with the CC2531 bootloader CC2530ZNP-SB.hex. There is a TAR file inside the TI Gateway download that includes the HEX file. I looked and this HEX file seems the same size as the others ~681k.

I can't seem to find any CC2538 USB dongles. So in that light I think the Z-Stack Home CC2531 firmware is the one to look at as it.

[presslab-us]

I see that the Z-Stack Home is at Z-Stack build 42930, compared to the Gateway build 38724.

I have flashed that build on my dongle. The error reported is:

14:37:17.990 [WARN ] [z.n.p.ZigBeeNetworkManagerImpl:1010 ] - ZDO_STARTUP_FROM_APP executed and timed out while waiting for response.
14:37:17.990 [ERROR] [z.n.p.ZigBeeNetworkManagerImpl:196 ] - Failed to start zigbee network.

And I cannot get anything to join. I am going to try and see what's going on.

[chris-hatton]

Hi Ryan,

Thanks for explaining about bin vs. hex, quite important to know that!

So maybe the Gateway package does, essentially, include the same Z-Stack
firmware?
That said, the 'Z-Stack Home' package does seems to be Ti's focus for
provision of the Z-Stack firmware, with it's hardware and security variants
and later version.

You're right about CC2538, that was a mis-paste on my part, I meant
CC2531ZNP-Pro.hex and variants.

Chris

On 6 March 2015 at 09:27, presslab-us [email protected] wrote:

I see that the Z-Stack Home is at Z-Stack build 42930, compared to the
Gateway build 38724.

—
Reply to this email directly or view it on GitHub
#13 (comment).

[presslab-us]

It seems like the CC2531 hex files are built for using the ZDO_MSG_CB_REGISTER API exclusively, which is contrary to what the readme file says. But this is how the TI Gateway works. (I captured the packets.)

I am not seeing ZDO responses until I enable the callback, and then they are the wrong type as expected. There are like 20 ZDO callbacks right now, I think it will be possible to convert from the ZDO callback to the normal ZDO response.

[tlaukkan]

Is anyone interested in taking a deeper look at this issue and has anyone been able to get one of new Z-Stack firmwares to work with zigbee4java?

[presslab-us]

I have it working since yesterday on the Home firmware Z-Stack 2.6.2 by translating the callback data. Binding doesn't work sometimes, this needs to be fixed and the code cleaned up.

[tlaukkan]

Wow, thats some nice progress :)

-Tommi
On Mar 7, 2015 4:05 PM, "presslab-us" [email protected] wrote:

I have it working since yesterday on the Home firmware Z-Stack 2.6.2 by
translating the callback data. Binding doesn't work sometimes, this needs
to be fixed and the code cleaned up.

—
Reply to this email directly or view it on GitHub
#13 (comment).

[presslab-us]

I have solved the attribute subscription problem, it's not related to the ZDO callbacks. For some reason the node is sending back extra bytes in the AF_INCOMING_MSG (which shouldn't matter), and the handler in zigbee4java is not using the specified length but the actual length. Not sure why I didn't see this with the other firmware but it does seem to be a bug in zigbee4java.

Now I am trying to make things cleaner and have a map for the class lookup. In Python I'd do something like this:
clusterToClass = {0x8001:ZDO_IEEE_ADDR_RSP, 0x8002:ZDO_NODE_DESC_RSP, *and so on*}
newPacket = clusterToClass[clusterId](frameData)
Where clusterId is the incoming cluster (0x8001, 0x8002, etc) and frameData is the argument to the constructor.
How could I do this cleanly in Java? I would do a map, but creating an object (with constructor arguments) from a class in a map seems difficult.

[tlaukkan]

Hi

Sounds like good progress continues :).

It would be nice if you could feed the changes upstream in smaller pieces. If you are planning larger refactoring please create first pull request including minimal changes required for supporting the new firmware. Then another one containing code changes related to refactoring. This will make review work much easier. Unless of course you are just refactoring some new code you wrote yourself.

Maybe something like this would work:
newPacket = idClassMap.get(id).getConstruct().newInstace()?

Best regards,
Tommi

[presslab-us]

Yes, I'm just refactoring the additions needed to support the new firmware. I had the numbers hardcoded to verify that it works and now I am cleaning it up for a pull request. The changes aren't that large really.

I will try your suggestion, thanks!

[chris-hatton]

@presslab-us I've been out of sync for a few days, but meaning to say; great work on debugging this issue! I suspected some kind of incompatibility after initial tests with the new firmware failed, but I'm not yet familiar enough with the ZigBee standard to be able to diagnose this in the way you did. Looking forward to merging and testing these changes as well. It would be really useful for me if you could briefly explain a bit further what Ti changed and speculate on why they did this? It sounded from your previous description like they have wrapped all calls in a callback container? Is this 'standard' and does it relate only to use with security modes?

@tlaukkan What is your take on the way the firmware interface has changed and, if the fix breaks compatibility with the old firmware, is this acceptable for zigbee4java, or should we be work towards supporting two modes? Personally I don't see a break to focus only on the new firmware as a bad thing: the new firmware is readily available and presumably contains other improvements and features from Ti.

@presslab-us I can help with the Java code for calling variable arguments constructors at runtime if your code would still benefit from it? Let me know and I can give an example, or we can arrange a Skype session to discuss. Cheers.

[presslab-us]

The ZDO_xxx_RSP commands are part of TI's ZTool interface. The firmware is not compiled with these commands implemented. I suspect it's because the ZDO_MSG_CB_REGISTER allows for multiple concurrent ZDO responses so this is now the default implementation. This would improve performance, maybe that's why TI did it that way.

As zigbee4java is written now, there is a lock on ZDO commands, so only one can be run at a time; there can be no concurrent ZDOs. The code I've added converts from the ZDO cluster response to a ZTool response so zigbee4java doesn't know the difference between the two. This means it should work with either firmware version.

I have the constructor working like so: Class<? extends ZToolPacket> newPacketClass = clusterToRSP.get(clusterId.get16BitValue()); and newPacket = (ZToolPacket) newPacketClass.asSubclass(ZToolPacket.class).getConstructor(int[].class).newInstance(frame); I would hope for something cleaner but I think that's the way it is.

[chris-hatton]

Thanks for explaining. That construction method is about as clean as it gets with more 'dynamic' Java I'm afraid... given that constructors cannot be included in interfaces. The only thing I would question is whether the '.asSubclass(ZToolPacket.class)' is necessary; the getConstructor should work just the same without it(?)

[presslab-us]

Yeah I didn't need that asSubclass. I had put it in there to get rid of a warning but I must have fixed it a different way because the warning didn't come back.

Here is my pull request: #16

[chris-hatton]

@presslab-us Thanks for that. I'll merge that locally and let you know how I go...
Will leave it for @tlaukkan to review & merge into main.

[tlaukkan]

Great job, I will review and merge latest this weekend. Chris, please let us know how this change works for you.

Best regars,
Tommi

[tlaukkan]

Hi

I am still lagging behind with this one. I took brief look and things look good but I need to find some time to flash one of my dongles to the new firmware to verify operation.

@presslab-us couple of questions:

1. is the coverage for the new firmware 100% or are only part of the operations provided by the firmware covered? Do we provide at least the same operations over serial as before?

2. What is the google-collections dependency for? I gather it should be guava and google-collections is obsolete.

Best regards,
Tommi

[chris-hatton]

Hi guys,

I seemed to spend the whole weekend trimming a giant hedge (not a euphemism), now I am aching.

Anyway, I have only had a chance to briefly try out the new firmware but I received 'bad packet' response from the CC2531 when running ZigbeeConsoleDesktop. @presslab-us is this something you have encountered or expect? Also, can you please confirm the .hex file you have flashed with, it's possible you have leap-frogged me with the version so perhaps I should be updating as well.

@tlaukkan You may have seen a few commits from me in the buildSupport branch. These are not ready for prime-time yet; it's been a learning process for me with Gradle and while the project does now build with Gradle, support profiles etc. I would like to give it a 2nd pass and restructure things a bit before merge to master. Getting there :-) I have been merging in all other changes along the way so there should be no 'merge shock' when it comes to re-integrating buildSupport with master.

@presslab-us @tlaukkan Re: The Google Collections dependency. I noticed the only place this was being used, was to create an ImmutableMap. Suggest we use the pure Java way of doing this which admittedly is a couple of lines more code but where that saves a whole dependency I think it is worth it. I've made this change in the buildSupport branch as an individual commit.

Time is short, hope to have some proper testing feedback for you in the next few days.
Keen to get it working :-)

Cheers,
Chris

[presslab-us]

@tlaukkan

1. I believe it has the same functionality with the new firmware as before. I went through the ZDO responses and added them all in. I did miss one that I found later (it had a different name) so it's possible I missed something else but I believe it has all the existing ZDO responses. Have you tested it with your older firmware?

2. Yes the google-collections is for the com.google.common.collect.ImmutableMap. A better way of doing this is fine by me! I have a lot to learn about Java...

@chris-hatton I did see the "bad packet" before I added https://github.com/presslab-us/zigbee4java/commit/41b19c3efd44e013405515af21034842cfc245fa and https://github.com/presslab-us/zigbee4java/commit/1619de90468e8ebed7786820a2f6c75702be92e0. Did you pull in these commits? I use the Z-Stack Home 1.2.2 CC2531ZNP-Pro-Secure_Standard.hex. If you still have problems post some of your log using TRACE.

Thanks,
Ryan

[presslab-us]

Now that I've added a bunch of devices (15 so far) I am sometimes getting a timeout on startup when it sends ZDO_STARTUP_FROM_APP. I have increased the timeout for just this command from 1s to 3s and it's fixed. How many devices have you guys joined? I'm thinking the timeout for this command should be larger by default.

[tlaukkan]

Hi, sounds like good idea to increase the timeout. I will try to run some tests with the old and new firmware today.

Best regards,
Tommi

[tlaukkan]

So far I have only tested with old firmware but it seems to work fine so I have merged these changes to master.

[tlaukkan]

I have added @presslab-us as contributor to this repository.

[tlaukkan]

I was able to flash the CC2531ZNP-Pro-Secure_Standard.hex to one of my dongles and it started up nicely with these changes from @presslab-us. I will need to try to join some devices to the network next.

[tlaukkan]

I have been able to join devices to network and command them succesfully. Nice work! 👍

[presslab-us]

Great! Thanks for testing it out and merging it. I feel a lot better knowing that we can use the latest firmware and won't be left behind on future updates.

[cdjackson]

Just as a matter of interest, is anyone having any problems with this firmware? I upgraded one of my sticks last night and am having a bunch of issues with it - quite a few timeouts and attribute read failures. I might change back to the old firmware to see if it goes away, but was just wondering if anyone else was having this issue (I'm guessing not, but just thought I'd mention it).
Chris

[cdjackson]

Actually, this might be related to the port speed - I'll raise another issue...

[chennakeshavaSoC]

Hi Guys,

I am looking for the right firmware and other configurations to see my CC2531 in action with openhab+zb4o. It seems my search is over here.

But is there any way to use this work with openhab framework?

Regards,
Chenna