Question: Implementation of mac commands

[brocaar]

Question: Implementation of mac commands

[ivajloip]

Actually when I started implementing this, it didn't seem as that much work, but after some time I realized this is not going to be the case. Anyway, it was to some extend interesting to write a little bit more Go code, as I am not very experienced with this language so for sure it was useful for me even only in this regard.

For the architecture, I tried to be inspired by the existing code + my sense of good practices. Basically my idea was to have some entities that can be shown to network server administrators for managing mac commands and that those entities will provide a small interface for loraserver.go to transfer and receive mac command requests and answers. It seemed necessary to do a little bit of refactoring in order the code to remain understandable (which contributes to some of the changes in loraserver.go). There is now a send method that receives a reply macPL, rxPacket + others and constructs two txPackets and sends them to the gateway. There is a method that creates the macPL for particular data that needs to be sent. There is a method for handling mac commands and some other helper methods.

The other changes are maybe less significant - nodeSession can store custom attributes for data rates and delays for a particular end-device (loraserver.go had to be changed accordingly - the new send method), app.js knows how to retrieve and update the entities, nodes.html now has very ugly views for the actual visualization.

The more important decisions from my point of view that I had to take were the methods that loraserver.go will access for mac commands manipulations and how/what to keep. Right now I keep the last sent set of mac commands + possibly their replies (if I have them) + mac commands to be sent (right now saved separately and sent only if the last mac commands that were sent are not the same or if we have not received a reply for them).

I hope it will not be that difficult for me to change my modifications

1. So that it will meet your architectural vision for the project.
2. So that it will be mergeable on the master branch.
3. So that it will be easily usable (maybe not exactly the case right now - if you don't know the spec well enough).

So far I have tried to provide a global overview of the change + give some reasons why I implemented all that. Please feel free to provide more details how you would expect this feature to be implemented or to ask for more details, which will make a code review feasible (I do realize it is not easy to see the change and understand it). Also I would not mind at all if I need to make some additional changes (rebase the branch on the new master for example, refactor some things, etc).

[brocaar]

Thanks! I won't merge as-is, since I still have to think what the right flow is for each MAC command and I would like to break the implementation of MAC commands into pieces. However, I'll take it as an example and make modifications where needed. This weekend I'll attend IoT Olympics and will try to get some feedback from other people on this.

[ivajloip]

I reviewed quickly the changes in for the mac commands and all looks good to me. Good job 👍

[brocaar]

I've just tagged version 0.8.0 with MAC command support :-)
http://loraserver.readthedocs.io/en/latest/changelog/#080
http://loraserver.readthedocs.io/en/latest/network-controller/

[brocaar]

Hi Ivaylo,

Thanks for looking into the MAC commands already! I have to dig a little bit deeper into the MAC commands before I can give a full answer, but as far as I think, there are three types: 1) initiated by the node, 2) initiated by the network server (e.g. frequency related MAC commands, could be based on settings?), 3) initiated by the user (give me the battery power, through the api).

I appreciate your help, but since some of the MAC commands are related to regions, I'm thinking about first implementing support for different ISM bands, or at least cleanup the currently hardcoded frequencies and put them a sane model that can be re-used for different ISM bands (e.g. by using different build flags, one could specify for which ISM band to compile).

As well I have to think about how to model which MAC commands are sent as FOpts (they can go together with an application payload) and which ones need to be sent as FRMPayload. This might be a (per application?) setting too.

Next week Thursday I'm going to present this project to the local Go meetup here in Amsterdam. Until then I'll cleanup some loose ends, fix some bugs and work on my presentation. So probably the week after I'll be working on new features again 😄

[ivajloip]

Hi Orne,

My implementation is almost ready and it seems to be working for most of the cases that I have tested (automatically and with my end-device). I have noticed though that there is some strange error when I was trying to send a couple of mac commands, when the bridge told me that it can not marshal an mac command that I did not sent. When I dug a little bit, I found that it calls txPacket.PHYPayload.MarshalBinary that on its turn, when FPort == 0 tries to marshal the FRMPayload as MAC commands. This fails as I have already encrypted the FRMPayload (as afterwards I don't have the encryption key), so inside FRMPayload there is only 1 DataPayload - the encrypted data. The simplest/cleanest way I see to fix this is to send the PHYPayload directly in binary (possibly base64). Let me know if I did not explain clearly enough the issue or if you have a better suggestion how it should be fixed.

[brocaar]

Thanks for your feedback, I think I know what is going wrong! I haven't started yet on the MAC commands yet. I'm currently doing some refactoring work on all three of the projects:

• lorawan: getting rid of the uplink argument and implementing a band sub-package containing band specific configurations
• loraserver: re-organizing the source so that vendoring becomes easier (no need to vendor the whole loraserver package)
• lora-semtech-bridge: send payloads as json over mqtt instead of gob (will add some more visibility)

After that has been completed I'll look into your issue!

[ivajloip]

Here is my implementation of the mac commands. It still needs some work, but it works fine at least for some of the cases (new channel, duty cycle, dev status have been tested with an end-device). I think that I am not going to change much my implementation , so you can review it. I only want to add the possibility to send mac commands over FOpts and I will make a pull request if all looks good to you. I might need to rework it in order to for it to match your changes in branch 0_5_dev.

[ivajloip]

Here is my implementation of the mac commands. It still needs some work, but it works fine at least for some of the cases (new channel, duty cycle, dev status have been tested with an end-device). I think that I am not going to change much my implementation , so you can review it. I only want to add the possibility to send mac commands over FOpts and I will make a pull request if all looks good to you. I might need to rework it in order to for it to match your changes in branch 0_5_dev.

[brocaar]

I've had a quick look at the diff: https://github.com/brocaar/loraserver/compare/master...Acklio:mac-commands?expand=1

First of all, thanks for your efforts!

However, the diff is huge. This makes it really hard to give good feedback and to shape it into a mergable state as it covers a lot of parts of the architecture. As well I'm a bit surprised by the amount of work that you already did. I would have preferred to discuss the right implementation and approach first.

[ivajloip]

I rebased the mac-commands branch on the current master (it seems to me that you have finished with the moving/restructuring). It is still a huge diff, so if you need more explanations, please don't hesitate to let me know :)

[brocaar]

I've started working on the MAC commands implementation. My idea is that the network-controller will be a component that is not part of LoRa Server and that LoRa Server provides various mqtt topics for communication. Outline of my idea:

There will be a new NetworkController backend which is responsible for the following topics:

application/[AppEUI]/node/[DevEUI]/rxinfo
Is already implemented (in the ApplicationBackend) but this will be moved to the NetworkController backend.

application/[AppEUI]/node/[DevEUI]/mac/rx
Sends received MAC commands to the network-controller

application/[AppEUI]/node/[DevEUI]/mac/tx
Topic to which the network-controller sends MAC commands to be transmitted to the node

application/[appEUI]/node/[DevEUI]/mac/error
Error topic.

LoRa Server will just keep a queue of MAC commands to be transmitted, but does not implement any MAC command logic so it doesn't know what to do with each MAC command. Example flow:

• network-controller sends a NewChannelReq to mac/tx
• LoRa Server transmits the MAC command when there is a downlink receive window to the node
• Node acks the MAC command
• LoRa Server forwards the ack to the network-controller
• The network controller then performs an RPC call to add the channel to the node-session

What do you think?

[ivajloip]

Hi Orne,

This generally seems as a good plan which allows a great deal of flexibility. I have some small concerns, but maybe they are due to my misunderstanding and in any case I don't think they are very serious. Here they are:

• MAC commands in the queue might overwrite each other. Maybe there should be a way to detect this or at least to cancel a MAC command.
• How are you going to handle LinkCheckReq? Through scheduling a reply (most-likely for some of the next uplinks)?
• What is the priority of MAC commands in comparison to the data down and do you plan to have a way to say - I want this MAC command in the FRMPayload? Might be useful.
• You might occasionally lose a message due to change in the configuration for data down that is not reflected on time (if I understood correctly that loraserver will not wait for a reply to MAC commands ack in order to proceed). This might arise if the RXDelay is changed or RX1DROffset is changed - the server will not send the data correctly in most of those cases.

All those questions could be resolved I think, I mention them here just to make sure they did not escape your attention. Otherwise this seems as a good way to go.

My 2c :)

[brocaar]

1. Yes, however I think it is best that LoRa Server should not know anything about MAC commands. The same issue applies I think for application payloads. Currently there is no way to cancel out a payload that is still in the queue. I think to start with, I'll just implement a FIFO queue. Maybe this can be resolved through the RPC API (eg. the network-controller makes an RPC call to remove a MAC command from the queue)?
2. When receiving an uplink MAC command, I think it is best to directly use the downlink receive window to sent the MAC response (if the network-controller responds in time to LoRa Server).
3. MAC commands can be combined with application payloads when FOpts is used. In any other case I think it makes sense to give MAC commands priority over application payloads (to make sure the node stays connected to the network e.g. when using ADR)? LoRa Server can indicate to the node that it should open an other receive window by setting FPending=true in the downlink packet. Regarding FRMPayload, that option is already there in the struct 😄 https://github.com/brocaar/loraserver/blob/network_controller_backend/models/packets.go#L83
4. In case the network-controller is requesting for example a channel change, I think it must wait for an ack from the node before modifying the node-session properties. Most / all ack payloads indicate if for example the data rate range was ok, channel frequency was ok etc... See for example the NewChannelAns payload.

Something for after the initial MAC command implementation: I'm planning to add an option to the API and web-interface to add proprietary MAC commands (0x80 to 0xFF range). Since all MAC commands are base64 encoded in the LoraServer <-> network-controller communication, I think the only thing needed to realise this is to configure a list of CID + optional payload size, so that LoRa Server is able to split the MAC commands correctly (this will require some changes at the https://github.com/brocaar/lorawan package too, but it will be quite straightforward).

If you're interested, see the pull-request link below for code changes so far. Feel free to comment and give feedback!

[ivajloip]

I still have not looked into your implementation, so I am replying only based on your comments and I will provide a second reply when I manage to take a look into the code.

1. Sounds good to me.
2. Ok, so loraserver should wait for some time before sending a reply (to give time for the network-controller to reply).
3. Sounds good :)
4. Yes, an acknowledgment is required, but my idea was if there is data that will be send to the end node on the downlink receive window right after an ACK was just received, then the loraserver need to wait for the RPC, otherwise the end node will have applied the change, but the loraserver will "notice" the change a little bit later.

Also the idea about proprietary MAC commands sounds very good to me 👍

[aminatamima]

Hi @brocaar ,
How did you implement the network-controller.md file please?
And in which file can i find how does my lora-gateway-bridge(or lora-server) do to receive a payload from a gateway?
Thank you in advance