simon622/mqtt-sn

MQTT-SN gateway and client implementations written in Java. Lightening fast. MQTT-SN is an optimised version of MQTT, designed for low powered sensor devices.

mqtt-sn's Issues

Configuration should be provided by yaml

It should be possible to pass a Yaml configuration file to the runtime as an argument to change the default behaviour

Implement the Protection Packet 0x1E

Implement the Protection Packet 0x1E https://www.oasis-open.org/apps/org/workgroup/mqtt-sn/documents.php

Use global scope properties in maven projects

Investigate the use of a parent project for dependancy management

Confirm session object encapsulates session state from CONNECT

The session object should be better represented in the APIs - presently the context object is used as a ket into the session registry - needs to be a first class citizen

Create version 2.0 alpha codecs

Per the technical committee track, need to create the codecs to prove out the version 2 functionality

Publish minus 1 options

Should be able to configure the runtime with 3 publish minus settings:

• ALLOW ALL
• ALLOW PREVIOUSLY AUTHORISED SESSION NETWORK ADDRESS
• DENY ALL

Payload integrity verification

-Integrity verification has been added to all protocol packets but not to payload.

RegAck handling throws a Classcast exception

When using Version 2.0 publishing QoS:2 the client sends a register event, on receipt of the RegAck the client tries to cast it to a V1.2 RegAck object.

The result is the following stack trace

java.lang.ClassCastException: class org.slj.mqtt.sn.wire.version2_0.payload.MqttsnRegack_V2_0 cannot be cast to class org.slj.mqtt.sn.wire.version1_2.payload.MqttsnRegack (org.slj.mqtt.sn.wire.version2_0.payload.MqttsnRegack_V2_0 and org.slj.mqtt.sn.wire.version1_2.payload.MqttsnRegack are in unnamed module of loader 'app')
at org.slj.mqtt.sn.impl.AbstractMqttsnMessageHandler.handleRegack(AbstractMqttsnMessageHandler.java:579)
at org.slj.mqtt.sn.impl.AbstractMqttsnMessageHandler.handle(AbstractMqttsnMessageHandler.java:360)
at org.slj.mqtt.sn.impl.AbstractMqttsnMessageHandler.receiveMessage(AbstractMqttsnMessageHandler.java:241)
at org.slj.mqtt.sn.impl.AbstractMqttsnTransport.receiveFromTransportInternal(AbstractMqttsnTransport.java:124)
at org.slj.mqtt.sn.impl.AbstractMqttsnTransport.lambda$receiveFromTransport$0(AbstractMqttsnTransport.java:62)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java)

Implement JWT based certificate signed clientIds

Create a new authentication provider which uses a token provided as part of the CONNECT clientId (which contains the actual clientId, date of token generation, token expiry) which is signed by a private key on the client and verified by a public certificate installed on the gateway as part of the session establishment.

Google IoT Core Gateway To Device Messaging

Ensure implicit subscriptions are created to enable state update events back to the device through the gateway

Client ID Provisioning Service

Should have a new service interface to allow dynamic generation of clientIds from either a clientId seed or null. x2 variants out of the box should be plainId and jwt from server certificate with clientId as seed

Support both versions of the protocol on same runtime

It is possible (using the version flag common across both versions of the protocol) to support both versions of the protocol in the same runtime. Need to create a wrapper codec to support this (the underlying implementation is already in place).

Use topic referencing

Invariably many subscription will use common topic filters. With thousands of devices and potentially many subs and regs per device there should be a topic ref service similar to the message store to ensure shared memory. Considered interning them but this will lead to leak over time as VM managed.

Performance of PathTriesTree during high volume subscribe to same node in tree

java.lang.Thread.State: RUNNABLE
at java.util.HashMap$HashIterator.nextNode([email protected]/HashMap.java:1601)
at java.util.HashMap$KeyIterator.next([email protected]/HashMap.java:1620)
at java.util.Collections$UnmodifiableCollection$1.next([email protected]/Collections.java:1054)
at java.util.AbstractCollection.addAll([email protected]/AbstractCollection.java:335)
at org.slj.mqtt.sn.utils.tree.PathTriesTree.searchTreeForMembers(PathTriesTree.java:234)
at org.slj.mqtt.sn.utils.tree.PathTriesTree.searchMembers(PathTriesTree.java:157)
at org.slj.mqtt.sn.utils.tree.PathTriesTree.hasMembers(PathTriesTree.java:161)

Zero Touch Provisioning Service

When the console is ready, should expose the use of the new ClientIdProvisioningService to generate valid Ids from the server certificate. The console should allow the registration of these allowed tokens from the console or from a text file upload.

Benchmark outbound serial dequeuing

Client -> Gateway messages (QoS 0,1,2) all dequeue quickly and appear to honour the minFlush time correctly. Gateway -> Client messages >= QoS 1 appear to dequeue with a small lag between each message.

Need to confirm the queue processor isn't backing off for the default backoff when serial dequeuing.

Allowing assignedClientId should be configurable

The ability in v2 to provide an empty clientId and have the gateway assign one should be configurable

Publish -1 and Publish Out of Band not working properly

Publish -1 and Publish Out of Band not working properly

Kafka connector

Need to create Kafka int

Implement QUIC transport type

Have an implementation of QUIC up and running in the gateway

The interactive CLIs should both take config to enable the gateway port to be configured

The client should take config to config a non standard remote port (need to confirm think its already done). The Gateway should use this config item to be the local port it listens on. (presently the reported port and bind port do not match)

Add network address white list regex

Should be able to configure the gateway to only allow connections from a network address that matches a regex (thus allowing ip range configuration)

Non unique network address attaches old session to new clientId

If a networkAddress is reused and an existing session is present for the old version of the address (potentially associated with a different cliendId) the old session is attached to the new CONNECT

Dropped TCP/IP connect auto reconnect loses subscriptions

Need to confirm the persistent connection on the broker side it either clean session false or reestabling subscriptions on auto reconnect mode

Implement a Tries tree implementation of the subscription registry

A tries tree implementation would provide a simple memory and time efficient algorithm for subscription storage.

Global dead letter queue

Need to add dead letter queue for visibility on global failures

Mqtt-sn client profiles

Sensor devices have common use cases, and sn is a slightly more complex lifecycle... define boiler plate use cases with sequence diagrams and example client code

Allow multiple transports to be running at the same time

The gateway should be able to be configured with multiple transport handlers in place to service different networks into the same runtime

Clean up disconnected sessions

Sessions which are transitioned to DISCONNECTED presently remain in memory forever. They should be tidied up after a configurable period of time to ensure the RAM version doesn't leak.

Unacknowledged redelivery on clean session false

reaped messages are currently requeued for ACTIVE delivery, need to support unacknowledged delivery on clean session false.

Implement IoT Azure connector

We have connectors for PAHO, AWS IoT, Google IoT Core. Id like to have one for Azure IoT as well. Initial look suggests no direct gateway metaphor al la GGL, so need to explore how to publish on behalf of.

Cluster support

Should have the ability to synchronise connections across VMs. The state that would need to be synchronised is as follows;

Client Id
Outbound messages queue
Subscriptions
Will data
Last seen
Session expiry interval
Max packet size
Retain behaviour (from CONNECT)

The cluster service gets notified upon new client side connection of the connecting clientId. If cleanStart = false then the state from existing session in the cluster should be synchronised to local connected gateway. If cleanStart is true the cluster still needs to be notified to clean up state held elsewhere.

If a connection is ACTIVE or SLEEPING on another node in the cluster, it should be terminated and removed after synchronisation occurs.

Expose will support to client & gateway

Activate the will support framework

DUP messages should be sent with the original msgId

Presently DUP messages are being sent with a new msgId which breaks conformance. This should be fixed to ensure the original msgId is used.

Confirmation to the runtime happens asynchronously to the last byte sent

Presently it is possible to send outbound QoS0 which will backup in the protocol job queue and mark messages sent that have not actually been sent by transport as are in back-pressure queue.

Need to move confirmation as sync to protocol send

Contention on network registry during concurrent additions and removals

"mqtt-sn-udp-transport-ingress-1771667101-1" #25 daemon prio=2 os_prio=31 cpu=8203.20ms elapsed=9.00s tid=0x000000013b811000 nid=0x5607 runnable [0x000000016e252000]
java.lang.Thread.State: RUNNABLE
at java.util.HashMap$HashIterator.nextNode([email protected]/HashMap.java:1601)
at java.util.HashMap$KeyIterator.next([email protected]/HashMap.java:1620)
at org.slj.mqtt.sn.net.NetworkAddressRegistry.removeExistingClientId(NetworkAddressRegistry.java:137)
- locked <0x00000007f8499dd8> (a java.util.Collections$SynchronizedMap)
at org.slj.mqtt.sn.impl.AbstractMqttsnMessageHandler.authorizeContext(AbstractMqttsnMessageHandler.java:70)
at org.slj.mqtt.sn.impl.AbstractMqttsnTransport.receiveFromTransportInternal(AbstractMqttsnTransport.java:80)
at org.slj.mqtt.sn.impl.AbstractTransport.lambda$receiveFromTransport$1(AbstractTransport.java:120)
at org.slj.mqtt.sn.impl.AbstractTransport$$Lambda$190/0x0000000800da5d08.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Executors.java:539)
at java.util.concurrent.FutureTask.run([email protected]/FutureTask.java:264)
at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1136)
at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:635)
at java.lang.Thread.run([email protected]/Thread.java:833)

Ping req should be optionally allowed to update network address

When a device floats network address with a pingreq and consistent clientId the gateway could optionally update its network registry with the new address. This should be configurable.

does not build -> Could not resolve dependencies for project org.slj:mqtt-sn-gateway:jar:0.2.0:

Hi,
try to build... as describe in README.md

Gateway Build

Git checkout the repository. For a simple standalone jar execution, run the following maven deps.
mvn -f mqtt-sn-codec clean install
mvn -f mqtt-sn-core clean install
-> OK so far.

mvn -f mqtt-sn-gateway clean package

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2.065 s
[INFO] Finished at: 2023-02-01T20:14:31+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project mqtt-sn-gateway: Could not resolve dependencies for project org.slj:mqtt-sn-gateway:jar:0.2.0: Failed to collect dependencies at org.slj:mqtt-sn-core:jar:0.2.0: Failed to read artifact descriptor for org.slj:mqtt-sn-core:jar:0.2.0: Could not find artifact org.slj:mqtt-sn:pom:0.2.0 in central (https://repo.maven.apache.org/maven2) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.

Any dear?
Thanx for help.
System:
Ubuntu 20.04.5 LTS \n \l
mvn --version
Apache Maven 3.6.3

Is there a maven repro availabel?

BR

Andreas

Using the CLI to publish to a non-registered short topic name fails silently

Using the CLI to publish to a non-registered short topic name fails silently

[Version 2 Codec] - Unsuback offsets incorrect

(Pending Confirmation). Reported by Matthew Buckton.

"The unsubAck decode has an issue and I have attached the fixed function

The offsets where out resulting in decode issues on unsubscribe Ack, this
would cause clients to disconnect even if they received a valid Ack from the
server. I have a job that now builds Simon's MQTT-SN client into artifactory
so I can redo the servers MQTT-SN tests to cover 1.2 and 2.0. Picked this up
in the subscription tests.

It's a simple fix, the msg Id starts at offset 2 and the return code starts
at offset 4."

Possible deadlock scenario

There is potential for a deadlock if client code attempts to shutdown the connection while the client is still attempting to connect.

If the thread created in MqttsnClient.activateManagedConnection() is executing it can acquire the connectionMonitor lock.
If a calling thread acquires a lock on the MqttsnClient instance and then attempts to call MqttsnClient.close(), this ends in a deadlock across connectionMonitor and MqttsnClient.this.

I recommend the class MqttsnClient does not lock on 'this', but rather an internal private field to avoid this situation

QoS 2 DUP deliveries results in duplicate messages being exposed to the runtime

When either the client or server sends a DUP delivery QoS, it is possible due to fine timing that the duplicate will be reported out of the state engine.

Short circuit connector

Create a connector which simply connects back to self and thus acts as a broker in its own right. This will allow for testing without third party round trip concerns.

Update logging framework

Move away from java util logging to logback etc.

Add network address black list

Add a configurable list of network addresses that will always be blocked by the gateway

Configurable integrity check field position

Raised by Matthew Buckton.

It should be possible to configure the position of the integrity field; options should be prefix / suffix (also applied to the infill variant).

Justification - Constrained devices can derive integrity more efficiently by calculating whilst parsing.

Disconnect failed retry devices

If multiple failures occur to a device, the reaper should consider disconnecting the device to save many active stale sessions. This should be configurable.

Session Radix Tree

The console should be able to bind into a new session listener (created and removed events) and register a RadixTree of the sessionIds resident on the run time to allow quick lookahead prefix searching

Mass client sending (70,000) messages at any QoS can yield processor stopping

Observed during soak test that a mass dequeue from client to server would result in the queue processing thread stopping after ~50 thousand messages - need to debug this