hotnoob/pythonprotocolgateway

Python Protocol Gateway reads data via Modbus RTU or other protocols and translates the data for MQTT. In the long run, Python Protocol Gateway will become a general purpose protocol gateway to translate between more than just modbus and mqtt. Growatt, EG4, Sigineer, SOK, PACE

For advanced configuration help, please checkout the wiki :)

Rebranding Again... last time.

if you installed this when it was called growatt2mqtt-hotnoob or InverterModBusToMQTT, you'll need to reinstall if you want to update.

Python Protocol Gateway

Python Protocol Gateway is a python-based service that reads data via Modbus RTU or other protocols and translates the data for MQTT. Configuration is handled via a small config files. In the long run, Python Protocol Gateway will become a general purpose protocol gateway to translate between more than just modbus and mqtt.

For specific device installation instructions please checkout the wiki: Growatt, EG4, Sigineer, SOK, PACE-BMS https://github.com/HotNoob/PythonProtocolGateway/wiki

General Installation

Connect the USB port on the inverter into your computer / device. This port is essentially modbus usb adapter. When connected, the device will show up as a serial port.

Alternatively, connect a usb adapter to your rs485 / can port with appropriate wiring.

install requirements

apt install pip python3 -y
pip install -r requirements.txt

python 3.9 or greater. python 3.10+ for best compatability.

Config file (config.cfg) - copy .example.cfg to .cfg

Edit configuration.

cp config.example.cfg  config.cfg
nano config.cfg

inverters / protocols

manually select protocol in .cfg protocol_version = {{version}}

v0.14 = growatt inverters 2020+
sigineer_v0.11 = sigineer inverters
growatt_2020_v1.24 = alt protocol for large growatt inverters - currently untested
eg4_v58 = eg4 inverters ( EG4-6000XP ) - confirmed working
srne_v3.9 = SRNE inverters - Untested
hdhk_16ch_ac_module = some chinese current monitoring device :P

more details on these protocols can be found in the wiki

run as script

python3 -u protocol_gateway.py

install as service

ppg can be used as a shorter service name ;)

cp protocol_gateway.example.service  /etc/systemd/system/protocol_gateway.service
nano /etc/systemd/system/protocol_gateway.service

edit working directory in service file to wherever you put the files

nano /etc/systemd/system/protocol_gateway.service

reload daemon, enable and start service

sudo systemctl daemon-reload
sudo systemctl enable protocol_gateway.service
sudo systemctl start protocol_gateway.service
systemctl status protocol_gateway.service

install mqtt on home assistant

Settings -> Add-Ons -> Add-On Store -> Mosquitto broker

setup Mosquitto broker

Settings -> People -> Users -> Add User -> Can only log in from the local network -> Fill Details

once installed; the device should show up on home assistant under mqtt

Settings -> Devices & Services -> MQTT

more docs on setting up mqtt here: https://www.home-assistant.io/integrations/mqtt i probably might have missed something. ha is new to me.

general update procedure

update files and restart script / service

git pull
systemctl restart protocol_gateway.service

**if you installed this when it was called growatt2mqtt-hotnoob or invertermodbustomqtt, you'll need to reinstall if you want to update. **

Unknown Status MQTT Home Assistant

If all values appear as "Unknown" This is a bug with home assistant's discovery that some times happens when adding for the first time. just restart the service / script and it will fix itself.

variable names

variable names have been modified for greater readability. if it's confusing you can change them via protocols/{version}_input_registry_map.csv you can also find the original documented variable names there; to use the original names, empty the variable name column the csvs are using ";" as the delimeter, because that is what open office uses.

variable_mask.txt

if you want to only send/get specific variables, put them in variable_mask.txt file. one variable per line. if list is empty all variables will be sent

variable1
variable2

variable_screen.txt

if you want to exclude specific variables, put them in the variable_screen.txt file. one variable per line.

variable_to_exclude
variable_to_exclude2

Any ModBus RTU Device

As i dive deeper into solar monitoring and general automation, i've come to the realization that ModBus RTU is the "standard" and basically... everything uses it. With how this is setup, it can be used with basically anything running ModBus RTU so long as you have the documentation.

So... don't mind me as i may add other devices such as battery bms' and... i have a home energy monitor on the way! so i'll be adding that when it arrives.

donate

this took me a while to make; and i had to make it because there werent any working solutions. donations would be appreciated.

(btc) bc1qh394vazcguedkw2rlklnuhapdq7qgpnnz9c3t0

Use Docker - untested

• docker build -t protocol_gateway
• docker run --device=/dev/ttyUSB0 protocol_gateway

pythonprotocolgateway's Issues

Requires paho-mqtt==1.6.1

Cannot run with paho-mqtt 2+, similar to adafruit bug linked below.
adafruit/Adafruit_IO_Python#152

mqtt json no attribute

Just tried setting json = true for mqtt

May 03 10:39:41 eg4monitor python3[1738]: INFO:classes.transports.transport_base:write data to mqtt transport
May 03 10:39:41 eg4monitor python3[1738]: INFO:classes.transports.transport_base:{'time_year': 24, 'time_month': 5, 'time_date': 3, 'time_hour': 10, 'time_minute': 39, 'time>
May 03 10:39:41 eg4monitor python3[1738]: Traceback (most recent call last):
May 03 10:39:41 eg4monitor python3[1738]:   File "/root/PythonProtocolGateway/protocol_gateway.py", line 200, in run
May 03 10:39:41 eg4monitor python3[1738]:     to_transport.write_data(info)
May 03 10:39:41 eg4monitor python3[1738]:   File "/root/PythonProtocolGateway/classes/transports/mqtt.py", line 173, in write_data
May 03 10:39:41 eg4monitor python3[1738]:     self.client.publish(self.base_topic, json_object, 0, properties=self.__properties)
May 03 10:39:41 eg4monitor python3[1738]:                                                                     ^^^^^^^^^^^^^^^^^
May 03 10:39:41 eg4monitor python3[1738]: AttributeError: 'mqtt' object has no attribute '_mqtt__properties'
May 03 10:39:41 eg4monitor python3[1738]: [2024-05-03 10:39:41,059]  {protocol_gateway.py:205}  ERROR - 'mqtt' object has no attribute '_mqtt__properties'
May 03 10:39:41 eg4monitor python3[1738]: ERROR:invertermodbustomqqt_log:'mqtt' object has no attribute '_mqtt__properties'
May 03 10:39:55 eg4monitor python3[1738]: get registers(0): 12 to 14 (3)

EG4 18kPV Support

Hello. I have an EG4 18Kpv and I would love to help this project in testing/creating compatibility to read the inverter/battery data locally. I have the RS232 eth -> USB cable that came with it. Do I need a different cable?

flexible bit value reading with template in csv column

some registers are holding sub-byte sized values. ie, 1bit, 2 bits, 4 bits in size.

so on the todo to add support for reading theese values.
i'm thinking maybe this as the format. in the csv. its a bit of a hack
{#bits:variable_name},{#bits:variable_name},... to be put into either the data_type column or documented name column.

or alternatively, could do it via register numbers and data type, with b4 being the starting bit, and data type defining the size.
ie,
data type: 4bits, register 70
data type: 2bits, register 70.b4
data type: 4bits, register 70.b6

leaning towards data_type and register, as its more clean.

Todo: SRNE inverter protocol v1.7

SRNE is fairly common; protocol is modbus. should be a quick addition.

TODO: Add "concatenate_registers" by data type

use data type to automatically create register ranges.
appears to be common pratice to rely on data type to specify the register ranges.

so, this will be required for the victron_gx3.3 protocol, since i don't want to manually specify all of the register ranges :P

will be for v1.1.3

todo: canbus reader / transport

when usb adapter comes in, i can start implementing canbus support.
protocol is similar to modbus rtu afaik.

TODO: custom serial frame reader / transport

work in progress for v1.0.9.

to implement custom frame-based serial reader, to allow for non-standard "frame" based "protocols" that utilize SOI and EOI bytes.
"frame" is for a lack of a better term. seems to be commonly used.

Requires Python 3.10 but does not check for version

If run with Python 3.9 (for example) the following error occurs.
TypeError: unsupported operand type(s) for |: 'type' and 'NoneType'

This is a problem for Raspberry Pi OS (at least 32-bit)

custom date parsing

Figured I'd share my code changes. I was frustrated for quite some time due to the SectionProxy config not allowing the mqtt class to read global settings.

root@eg4monitor:~/PythonProtocolGateway# diff -ur classes/transports/mqtt.py classes/transports/mqtt_custom.py
--- classes/transports/mqtt.py	2024-05-03 10:42:57.881082043 -0500
+++ classes/transports/mqtt_custom.py	2024-05-03 14:05:28.387262768 -0500
@@ -4,6 +4,7 @@
 import time
 import json
 import warnings
+import datetime

 import paho.mqtt.client
 import paho.mqtt.properties
@@ -17,7 +18,7 @@
 from ..protocol_settings import registry_map_entry, WriteMode, Registry_Type


-class mqtt(transport_base):
+class mqtt_custom(transport_base):
     ''' for future; this will hold mqtt transport'''
     host : str
     port : int = 1883
@@ -57,6 +58,12 @@
         self.json = strtobool(settings.get('json', self.json))
         self.reconnect_delay = settings.getint('reconnect_delay', fallback=7)
         self.max_precision = settings.getint('max_precision', fallback=self.max_precision)
+        self.combine_timestamp = settings.getboolean('combine_timestamp', fallback=False)
+
+        # Timestamp mapping config
+        self.timestamp_map = {}
+        for i in ['year', 'month', 'date', 'hour', 'minute', 'second']:
+            self.timestamp_map[i] = settings.get(i)

         if not isinstance( self.reconnect_delay , int) or self.reconnect_delay < 1: #minumum 1 second
             self.reconnect_delay = 1
@@ -162,11 +169,34 @@

         self._log.info("write data to mqtt transport")
         self._log.info(data)
+
         #have to send this every loop, because mqtt doesnt disconnect when HA restarts. HA bug.
         info = self.client.publish(self.base_topic + "/availability","online", qos=0,retain=True)
         if info.rc == MQTT_ERR_NO_CONN:
             self.connected = False

+        if self.combine_timestamp:
+            """ Some inverters provide a timestamp, but it is split into multiple data points.
+                This functionality allows for creating a new timestamp data point, using the
+                documented-name from the registry maps.
+            """
+            tmap = self.timestamp_map
+
+            # Construct timestamp string ISO-8601
+            year = data[tmap['year']]
+            if year < 50:
+                year += 2000
+
+            ts = datetime.datetime(year, data[tmap['month']], data[tmap['date']],
+                data[tmap['hour']], data[tmap['minute']], data[tmap['second']])
+
+            # Add new timesstamp to data
+            data["timestamp"] = ts.isoformat()
+
+            # Remove timestamp parts from original data dict
+            for k in tmap:
+                del(data[tmap[k]])
+
         if(self.json):
             # Serializing json
             json_object = json.dumps(data, indent=4)

And the added sections in config.cfg

[transport.1]
...
json = true
...
# Do you want a single timestamp created?
combine_timestamp = true

# Timestamp mapping for combine_timestamp
# Use the documented_name from the protocol
# registry map to create a mapping.
year = time_year
month = time_month
date = time_date
hour = time_hour
minute = time_minute
second = time_second

Which generates MQTT events:

/home/inverter {
    "pv1_voltage": 176.9,
    "pv2_voltage": 192.20000000000002,
    "pv3_voltage": 5.800000000000001,
    "soc": 48,
    "ppv1": 301.0,
    "ppv2": 83.0,
    "ppv3": 0.0,
    "pcharge": 269.0,
    "pdischarge": 0.0,
    "peps_l1n": 44.0,
    "peps_l2n": 64.0,
    "timestamp": "2024-05-03T14:47:00"
}

and now I can parse this in Telegraf using the json datatype to send to InfluxDB.

Cleanup EG4 holding registery map for write access

with the popularity of eg4 inverters, should clean up the holding register csv so that write access can be potentailly enabled.

still on todo to create a manual safety verification process to enable writing.

protocol analyzer - value range checking - flag code csv format

so... looks like its not good enough to check for zeros :P

i'm going to have to clean up the "values" columns in the csvs, and come up with a standard format / implementation for checking the values.

i have to think abit about this. currently flags/codes are done via json which i prefer as it is cleaner.
maybe i'll add a flag_set={flag_name} sort of thing.

Proper way to write to registers

What's the correct way to write to a register? I'm not seeing much documentation on this. I just want to write to the registers holding the clock on my inverter to correct the time drift (and probably DST in Nov). I'm thinking this be just a one-shot type script that I can call on a cron job, say every Sunday, to set the clock. I'm not understanding how after I set 'write = true', then where do I specify which regs and what values to write? Thanks.

TODO: protocol based default settings

add default settings config for protocols to the .json file

ie, baud rate, sleep, max batch size ect..

since protocols will match similar devices.
lessen the learning curve.

Need No Read Ranges

could be a pain to add, but need to add the ability to forcefully exclude specific registers from the reading ranges.

some devices, such as my sigineer inverter will bug out / not reply when certain ranges are read, because they are "reserved" or protected from reading.

Todo: dynamic entires for register ranges ect...

this is especially needed for pylon rs485 protocol. some other protocols could benefit as well.

step 1. post pone loading config for unloadable dynamic registers ( and commands )
step 2. do a full scan and loop until completed, or more complicated, identify required "variables" and load.
step 3. apply skipped entries to memory / register list.

down side is, this initially doesnt allow for further dynamic loading. ie if a value changes... could be added later i guess.

the pending csv rows can be stored in a variable / memory, and edited on the fly as variables arrive.

for a format, can maybe do {{variable_name}}
or, [[variable_name]] to avoid json conflicts.

example usage:

document_name = cell_count

documented_name = battery_1_cell_[0-[cell_count]]
register = x4642.[(0-[cell_count])+1]

shit... going to need to implement a basic math implementation as well... ranges will have to be done exclusively with ~ when within [] which will trigger "maths"

TODO: modbus sniffer

create a read only "reader" / "transport"

the purpose is sniff data when not master. since modbus rtu only allows for 1 master in many cases'