Version 3.0 Discussion

[theagentk]

Version 3.0 Discussion

[tsightler]

I've continue to push updates to the dev tree. Currently it's working pretty well with simple devices like switches, sockets and dimmers, and it works fine with other devices I've tested as well. You can now monitor and set any and all DPS values using

Tuya JSON

tuya/<device_id>/dps/state
tuya/<device_id>/dps/command

These topics allow you to set and get raw Tuya JSON. This was previously possible with just /dps, but now requires /dps/command topic. You can set multiple topics at once, etc., just like previous versions.

Simple String Data

tuya/<device_id>/dps/<#>/state
tuya/<device_id>/dps/<#>/command

If you don't want to use JSON you can also get and set individual values from each DPS key using these topics. While it was always possible to read these, the development version now allows you to set values with these topics as well so, for example, a simple dimmer switch might look like this:

tuya/<device_id>/dps/1/state = true/false <- Switch on/off state
tuya/<device_id>/dps/2/state = 0-255 <- Brightness

To change brightness state simply send "100" message to tuya/<device_id>/dps/2/command, so even with simple mapping you can create, for example, a dimmer switch (light component) in Home Assistant.

Since you can access and set and DPS value, you can use tools like MQTT explorer monitor DPS values while you change settings in the Tuya/SmartLife app and quicly learn values.

The current branch also implements the first concept of templating using a simple JSON model, for example, a simple device like the dimmer above might look like this:

{
    "state": { "dpsKey": 1, "dpsType": "bool" },
    "brightness_state": { "dpsKey": 2, "dpsType": "int", "minVal": 0, "maxVal": 255 }
}

"bool" devices show "ON" or "OFF based on true/false (in the future I'll allow to set the desired bool translation) and the corresponding command topic is automatically enable and will accept simple "true/false", "on/off", and "0/1" to change the state.

So with that simple template you end up with a device like this

tuya/<device_id>/state = ON/OFF <- Switch state
tuya/<device_id>/command <- Set switch state with (0/1, true/false, on/off)
tuya/<device_id>/brightness_state = 0-255 <- Brightness
tuya/<device_id>/brightness_command <- 0-255 <- Set brightness (other values will be logged as out of range

Currently the code tries to guess the device type based on the returned DPS values and their values, but right now it's only simple switches and dimmers. You can specify a device template by modifying the devices.conf file and adding type: "dimmer" or type: "switch" (the format is JSON5 so the syntax is more forgiving than regular JSON). I hope to grow the available pre-defined templates over time.

I still need to work on transforms, mainly for colors, since the Tuya color format doesn't map directly to what most home automation platforms expect out of the box, but there are a lot of different lights and I only have one so I may need some help there.

After some of that works done it's pretty easy to build MQTT discovery topics for Home Assistant (and others that support Home Assistant MQTT discovery protocol, like OpenHAB).

Overall I think this is a solid base for 3.0 for tuya-mqtt. For the simplest of case with just switches you can mostly just take the output from tuya-cli wizard, save it as devices.conf, and fire it up. For more advanced cases you'll have to map DPS values.

Hopefully over the next few weeks I'll finish the template engine and leads get the color transform working, but I'm already using it for most of my devices with good success.

[tsightler]

Yep, just add ip and version (for protocol version, either 3.1 or 3.3) options to your devices.conf like the below example:

  {
    name: 'Your Device',
    id: '86435357d80123456789',
    key: '8b2a699876543210',
    ip: '192.168.10.10',
    version: '3.3'
  }

[Flip76]

Yes, now it works :-)

[tsightler]

I don't think I will go that far having to have the same device mulitple times, and it's doesn't map to the device model of most home automation systems very well. Most home automations systems use some concept of a device with capabilities (Home Assistant uses device/entitiy, OpenHAB uses things/channels), the idea being that the "device" represents the physical device and the "capability" represents the individual components of that device that can be controlled or monitored.

I like to follow that same structure in the topic model so that it maps easily while following the above you would end up with three different "devices", rather than a single device with multiple control points. However, with beta2 you will be able to get very close to your goal with the generic device using something simple like this:

{
  name: 'Steckdosenleiste',
  id: '23804861d8bfXXXXXXXX',
  key: '868c7fa1XXXXXXXX',
  ip: '192.168.103.15',
  version: '3.3',
  template: {
    monitor1_state: {
      key: 1,
      type: 'bool'
    },
    monitor2_state: { 
      key: 2,
      type: 'bool'
    }
  }
}

So then you can control devices with friendly topics like:

tuya/steckdosenleiste/monitor1_command
tuya/steckdosenleiste/monitor2_command

I think that's pretty reasonable and maps to the device model cleanly.

[tsightler]

Pushed 3.0-beta4. Adds support for heartbeat (to report device online/offline via /status topic). Includes a lot of (mostly) minor fixes to device handling, adds support for color temperature. I've acquired a few more bulbs with color temperature and also with both old and new style HSB/HSBHEX color and settings so I've been able to test a wider variety of bulbs. Overall things seems to be working quite well although there are still a few behaviors I don't like that I believe I can improve.

Still have a handful of things to add, and then lots of documentation to update, but this code it working very well for me over the last week.

I'm still looking for some feedback on the RGBTW bulb support around how to handle white/color mode and brightness. The Tuya bulbs have separate brightness for white vs color mode (you can see this in the tuya app), which is not uncommon, especially for bulbs that are much brighter in white mode than in color mode (like most Tuya bulbs). Below is the current behavior:

White mode

1. White mode brightess is reported/controlled by white_value_state/command topic
2. Color temperature is reported/controlled by color_temp_state/command topic

Color mode
Color mode actually provides 3 topics:

1. Color mode brightness is reported/controlled by brightness_state/command topic
2. Color mode Hue/Saturation is reported/controlled by hs_state/command topic
3. Color mode Hue/Saturation/Brightness is reported/controlled by hsb_state/command topic

Why three topics with overlapping controls? Some automation platforms, like Home Assistant, require separate topics for brightness vs hue/saturation. I'm not sure why, but probably it makes it easier to control brightness alone vs always needing to send all three components, or maybe it's because some bulbs use a single brightness control for both color/white mode so having it separate makes sense. However, other platforms, like OpenHAB, appear to strongly prefer a single topic with all three components comma separated. Offering both makes it easy to consume this integration with either platform.

Switching between white/color mode works as follows:

1. Any time white level (white brightness) or color temp are changed, switched to white mode
2. Any time saturation is below 10%, switches to white mode
3. If hue is changed, and saturation is above 10%, switch to color mode
4. If saturation is increased above 10%, switch to color mode
5. If color mode hue/brightness is changed, but saturation remains below 10%, stay in white mode

I feel like the above is reasonable logic, simple and the code is easy/straightforward (there's something to be said for this), but it causes some behaviors that I don't really like. For example, some bulbs automatically switch to color mode if you change any color value, so, if you change hue or color brightness, even if the saturation is <10%, you'll get a quick flicker while the bulb changes to color mode and is immediately flipped to white mode. I admittedly think this will be pretty uncommon, but it just bugs me.

Another issue is that some bulbs completely ignore any changes to the color attributes when the bulb is in white mode. So, for example, if you change the color brightness value while the saturation is <10%, so the bulb is in white mode, and then later switch to color mode, the brightness will still be the old value. Other bulbs always accept changes to any value, and only switch modes on manual command to do so, which is, IMO, the best behavior.

So, the issue is that it's difficult to get consistent behavior because different bulbs behave differently and I have no way to know what bulbs behave in what way.

The code currently attempts to somewhat compensate for this by caching the HSB color state while the bulb is in white mode and basically "faking" the updates of colors, storing them for a future switch to color mode. It complicates the code, but gives reasonably consistent behavior across all bulb types.

Currently, since I'm not sure how common the use of "white level" for white brightness is outside of Home Assistant, I'm thinking I will rename the brightness topics to

color_brightness_state/command
white_brightness_state/command

Feedback is welcome.

[Trauma]

I've just tested the last version with a version 3.3 RGBTWLight which is branded LSC Smart Connect LED with Home Assistant, and it just work really fine !

And i have to say that you're white/color switching implementation just feels really natural to me. Great job ! Actually tuya-mqtt implementation is way better to me than the cloud tuya api integration (or at least it works better for this bulb).

About the topics naming, since I stumble on understanding it's meaning in the past, i'll up vote that too. But in a "it works out of the box" experience it doesn't matter that much ;)

I've actually noticed only one strange behavior in home assistant, when switching from color to white mode : the bulb icon stick to the latest known color, and it do not seems to change unless you manually pick a white color in the color picker. I do not know how mqtt light integration handle this, i've seen you've added a mode_state which seems to be intended for this purpose, but could not find any occurence in HA doc.

[Trauma]

Forget about the connection issues, it is indeed a disconnected device. 👍

[tsightler]

Thanks for confirming, I've been wracking my brain trying to figure out what else I could have missed. I've really put a lot of effort into testing the failure cases because I believe home automation must be near 100% reliable to not be more trouble than it's worth and to keep the wife acceptance factor high. That's one thing I learned when building ring-mqtt, it needs to handle and recover automatically from pretty much all failure cases. I'm not sure tuya-mqtt is quite there, but I've spent a lot of time testing things like powering off devices, restarting devices, rebooting APs for devices, blocking devices from APs, forcing IP changes, etc, because I expect tuya-mqtt to recover from all of those cases. The current versions pass all of my tests, but perhaps my tests are not fully comprehensive...yet. Long term reliability is more difficult to test.

[tsightler]

Yeah, I noticed this bug about a week ago, status is only sent during device connect/disconnect event, not during publish/republish. I probably should force send it during republish. I have a small backlog of bug fixes/improvements that I'll probably try to group together into a 3.0.1 sometime soon which will include a fix for this.

[tsightler]

Hmm...maybe none of this is necessary as it seems most of this is already implemented by the tuyagateway project. Perhaps it's time to freeze and retire this project.

[thomasdannenmuller]

Hmm...maybe none of this is necessary as it seems most of this is already implemented by the tuyagateway project. Perhaps it's time to freeze and retire this project.

Interesting found !

[thomasdannenmuller]

I giving it a try right now. While the architecture seems fine to me, the configuration of gismocaster is a kind of nightmare, but it looks powerful. The docker integration has to be reworked, it seems ok for tuyagateway, but gismocatcher is not.

Your idea to build the discovery topics from tuya-cli output seems much more smarter to me.

[tsightler]

I saw a lot of issues around gismocaster on the project page so that definitely seems to be a pain point. Regardless, most people seem to be able to get it to work eventually, and it seems that part would be easier to fix vs building a different tool completely. It looks like gismocaster just stores it's config information in MQTT via retain, which I'm not a big fan of doing as I find people have a tendency to treat their MQTT broker as throwaway, they'll remove/reinstall it, etc.

That being said, it seems like a very active project and I don't know if it's a good idea to try to compete with it. Maybe I should just look to create a simple command line tool that builds a gismocaster compatible config in MQTT from the tuya-cli output.

Thanks for testing it out, I might give it a spin myself as well before making a decision whether to go forward with this project or not.

[tsightler]

So I've decided to go ahead and attempt a 3.0 version of this project, even with tuyagateway being already available. I've pushed the first set of features to the dev branch which does the following:

• Drops support for the old topic based config in favor of reading the file "devices.conf" in the same format as output from "tuya-cli wizard" during startup
• By default devices.conf should have "name", "id" and "key" for each device, so script will use automatic discovery which will find device on network, and discover protocol version where possible. If you want to manually specific IP address of device you can edit file and add "ip" and "version" parameters, with version being 3.1 or 3.3 (default is 3.1 if not specified).
• Shortened device topic to tuya/<device_name_or_id>/. Example if device with name="Kitchen Lights" is tuya/kitchen_lights/state or if name is not available then tuya/<device_id>/state
• Bump all package versions to latest including TuyAPI 5.3.2 which fixes a discovery bug, if upgrading be sure to run "npm i" to upgrade all dependenies.

This has allowed me to remove a lot of code and simplify the processing of MQTT topics so now I'm going to work on that part. Currently the plan is to attempt to detect switches/dimmers/lights based on common DPS values and, if the profile does not match, simply default to only publishing DPS values, which can be used with pretty much any device. I've even thought about implementing some basic templating (DPS to capability mapping) so users could contribute working maps. After that I'll implement some basic MQTT discovery for known devices.

Admittedly, I'm not sure if this project will be easier to use than tuyagateway, or really offer any benefits over it, but the slightly different approach to device configuration might appeal to some, especially those using the project for non-Home Assistant cases.

[thomasdannenmuller]

Drops support for the old topic based config in favor of reading the file "devices.conf" in the same format as output from "tuya-cli wizard" during startup By default devices.conf should have "name", "id" and "key" for each device, so script will use automatic discovery which will find device on network, and discover protocol version where possible. If you want to manually specific IP address of device you can edit file and add "ip" and "version" parameters, with version being 3.1 or 3.3 (default is 3.1 if not specified).

When i first came to tuya-mqtt i though it could be cool to have that. I've build the same data structure to store all the necessary to build the mqtt topics, and issue commands, So i'll naturally say go for this.

Shortened device topic to tuya/<device_name_or_id>/. Example if device with name="Kitchen Lights" is tuya/kitchen_lights/state or if name is not available then tuya/<device_id>/state

Sounds good too, like this we will have far nicer topic tree.

According tuyagateway : While gismocaster brings a good way to build the HA discovery conf, it's really a pain to setup. I've encontered lots of troubles to make it work, and dispite it's allowing to build discovery for lots off device classes I guess that on the tuyagateway side lot of work reamain to be done. To me tuya-mqtt as better coverage off the tuyapi to mqtt interface.

I can't say for now if i'dd prefer to use a gismocaster like tool or configuring myself HA. I guess the zigbee2mqtt like design you propose is way better. 👍

[Flip76]

I just opened my toy box and I still have a lot of different tuya devices (light bulbs, switches, sockets, gateway, IR-remote control, etc.) - looking forward to test it with the 3.0 version :-)

[tsightler]

You can always pull the dev branch right now, it's poorly documented, but it does work. I'm actively using it for integration with one socket, a dimmer that won't flash Tasmota (so far), and RGBW light (although the light only works for on/off right now).

That being said, after wrangling with the code as it was for quite a while, and trying to just overlay the changes I had planned over it, I gave up and decided on a near complete rewrite. I looked at a number of other projects, including tuyagateway, but it seemed not very usable when I tried it, so I decided to pretty much follow the model of the homebridge-tuya. They already use a similar configuration concept (modified devices file) and I really like the way they split supported devices into their own files. It very much mirrors the model I used for ring-mqtt, for which I used the homebridge-ring project as my guide.

Based on this I started a complete refactor and already have it working with my switch and dimmer, and am now working on the light and a "custom" component where a user can build their own topic mapping (for advanced users only). The new code is simpler for me as a novice coder, to follow and each device now has it's own data handler so no complex logic or event handlers feeding data into the main code, etc.

I hope to get this in the next few days and push 3.0-beta2. I certainly won't be able to support as many devices as homebridge-tuya right off the bat, but at least the framework for adding new devices will be easy and the "custom" device will still allow people to build their own or access the saw DPS json/values.

[Flip76]

Actually I am using hombridge-tuya too but I've got several issues with it (when device get's switched off, it remains on in homekit and the massive logging is slowing down the whole homebridge-system)... But the handling is quite easy and straight forward. I also tried tuyagateway but I didn't get it to work properly...

So you wrote ring-mqtt too? I am currently using an older OH-plugin. But in order to get a real-time notification in OH, I exposed a switch to Amazon Echo and defined a rule which enables the exposed switch in realtime when somebody rings the bell. Unfortunately I have to wait several minutes until the video is available at ring servers. Then I download it and with ffmpeg I cut out a single picture which I send to my messenger.... Would ring-mqtt announce a "ding" in real-time?

Will clone the dev branch in a minute :-)

[tsightler]

Yeah, ring-mqtt is my project from the start, while tuya-mqtt is a project written by a previous author and I'm just a maintainer. Unfortunately there's no true "real time" with Ring cameras as the API for cameras only supports polling, however, the polling interval is 2 seconds (same as the Ring app) so it's usually fairly fast. However, ring-mqtt doesn't support any video so you'd have to obtain that via other means, which I guess you are already doing.

Anyway, I guess that's pretty offtopic for this thread. I thought I was making really good progress with tuya-mqtt, but I've hit quite a snag. Apparently with some of the newer devices it's not possible to get the initial state of the device with the get() call, you just receive an invalid response and I'm not sure there's any workaround for this in TuyAPI, at least, I haven't figured one out yet. It's still possible to control the device and state updates are sent on state changes, but nobody wants to have to toggle all of the settings just to get state. I see some workarounds in some of the other Tuya libraries, but none of them seem to work reliably, or maybe I just don' t understand them enough to implement them correctly.

If I can overcome this issue, I think I'm probably only a few hours of hacking away from being able to get a 3.0-beta2 release as otherwise the code seems to work well enough, but not getting the initial state makes the overall solution not viable from my perspective.

[tsightler]

Made a lot of progress tonight, figured out a way around the initial state issue and I have it working quite well for the devices I have access to (admittedly only a few). Have Home Assistant discovery working quite well, full color working for the light. I only have 3 specific device files written so far (swtich/socket, dimmer, RGBTW light, which are the devices I have), and I'm going to build a "generic" device which will allow access to device states/command via DPS topics directly and, in the future, hopefully a simple template based model for mapping DPS values to friendly topics.

Once those tasks are done I'll upload a new 3.0 beta and hopefully somebody can test it out on a wider set of devices. On the topic of devices, homebridge-tuya has device specific support for more than a dozen devices. It will probably be quite a while before I can add support for anywhere near that many, so I'd be curious if there is a priority of devices that people would like to see support for, especially knowing that I don't have those devices.

[Flip76]

I will setup all my Tuya devices over the weekend and let it run. Meanwhile I had a first try yesterday with the dev-branch. Unfortunately this didn't work:

mosquitto_pub -t tuya/23804861d8bfXXXXXXXX/dps/2/command -m ON

nor this:

mosquitto_pub -t tuya/23804861d8bfXXXXXXXX/868c7fa1XXXXXXXX/192.168.103.15/dps/2/command -m ON

I'll pull the latest changes and try it again

[tsightler]

Were you able to see state changes for the device at tuya/23804861d8bfXXXXXXXX/dps/2/state?

Also, most of the tuya devices I've seen require "true/false", not ON/OFF. Other than manually mapping with a schema there's no way to know (that I'm aware of) if any give DPS is a string, a number, or a boolean, so there is no processing on command messages sent to DPS indexes, you must send the command exactly as the DPS expects. If it uses true/false you must send true/false.

For example, on my simple dimmer, it's on/off control is DPS 1, if I want to use DPS to control it I have to send:

mosquitto_pub -t tuya/23804861d8bfXXXXXXXX/dps/1/command -m true

Certainly the second command would not work. Also, if you have a friendly name in your devices.conf file then the topic will be the friendly name, converted to all lower case, with spaces converted to underscores. So, for example, if your devices.conf looks like this:

[
  {
    "name": "Kitchen Table",
    "id": "86435357d8b87654321",
    "key": "8b2a69c61234567"
  }
]

Then the topic would be tuya/kitchen_table/dps/2/command, however, this should be very easy to see by manipulating the device via the Tuya/SmartThings app and seeing what topics are displayed.

In the new beta, you will be able to configure supported devices and their DPS values (if different from defaults) or, for the custom device, define a set of DPS values that should be queried for state during startup so you will be able to see the state topics without manipulating them elsewhere.

Basically, the summary version is that, when using the DPS topics directly, the code is not going to help you with any friendly transforms, you have to know exactly what format for values that DPS expects and send exactly that. You can even have more control by using the JSON command topic.

Normally, devices map values to "friendly" topics, so, for example, while my switch above sends and requires true/false on DPS1, I'd prefer to send and receive "on/off" since that's what Home Assistant expects by default (although of course it can be changed). When I map the value to a "friendly" topic I simply tell it that DPS/1 is a "bool" and now, any values going to/from DPS 1 via that friendly topic will also be transformed into ON/OFF and you can send ON/OFF, 0/1, true/false as commands and, since the code knows it's a "bool" it will translate them into true/false on the backend. In beta 2, you'll (hopefully) be able to create a simple template for unknown devices to perform "DPS" to "Friendly" topic mapping. For example, here's what a template might look like for my simple dimmer:

state: {
    key: this.config.dpsPower,
    type: 'bool'
},
brightness_state: { 
    key: this.config.dpsBrightness,
    type: 'int',
    min: (this.config.brightnessScale = 1000) ? 10 : 1,
    max: this.config.brightnessScale,
    scale: this.config.brightnessScale
}

The above says that DPS value 1 is a bool value and should be mapped as such to /state topic and brightness is an interger, with value values between 10-1000. So, before mapping I could control my device with:

tuya/<device_name>/DPS/1/state <- shows true/false for On/Off
tuya/<device_name>/DPS/1/command <- required true/false
tuya/<device_name>/DPS/2/command <- required int between 10-1000, but other values caused unpredictable behavior

After mapping I could still use the DPS values if I want, but instead I can use (an equivalent "command" topic is created for all mapped "state" topics automatically":

tuya/<device_name>/state <- shows ON/OFF
tuya/<device_name>/command <- now accepts "ON/OFF, "0/1" or "true/false" as well as "toggle" to flip state
tuya/<device_name>/brightness_command <- now accepts only integers 10-1000

This allows you to create a simple template for any device. Of course a template is not required, you can always manipulate DPS values directly, but templates can be more forgiving. Also, some DPS values aren't that usable without some transforming, for example, the color values of a light bulb. Tuya uses a hex format that needs to be translated for most tools to be able to easily consume it. Of course, you could translate it in Node Red, or in the home automation templating engine, but I've written helper functions that allow translating and mapping to "friendly" topics that are already in a format expected by common Home Automation systems.

[Flip76]

Ah ok, I missed that! I thougt ON/OFF will be automatically translated... The command for the current master branch which works is:

mosquitto_pub -t tuya/ver3.3/23804861d8bfXXXXXXXX/868c7fa1XXXXXXXX/192.168.103.15/command -m "{ \"dps\": 1, \"set\": true }"

Do you get the protocol version automatically or do I still have to put it somewhere?

[Flip76]

Unfortunately it doesn't work:

The command I used:

mosquitto_pub -t tuya/23804861d8bfXXXXXXXX/dps/1/command -m true

The devices.conf looks like:

{
name: 'Steckdosenleiste',
id: '23804861d8bXXXXXXXX',
key: '868c7fa1XXXXXXXX'
},

Instead this still works with the dev-branch:

mosquitto_pub -t tuya/ver3.3/23804861d8bfXXXXXXXX/868c7fa1XXXXXXXX/192.168.103.15/command -m "{ \"dps\": 1, \"set\": true }"

[Flip76]

Oh hang on! Look like the problem is between my ears :)
I just did a git clone but I forgot to add the branch :-)

git clone --single-branch --branch dev https://github.com/TheAgentK/tuya-mqtt.git

git branch

• dev

Will try again now :-)

[Flip76]

So I tried now:

mosquitto_pub -t tuya/steckdosenleiste/dps/1/command -m true

and the debug shows:

  TuyAPI:mqtt Receive settings {"topic":"tuya/steckdosenleiste/dps/1/command","message":"true"} +5s
  TuyAPI:mqtt Received command for DPS1:  true +2ms
  TuyAPI:device set: { dps: '1', set: true } +5s
  TuyAPI SET Payload: +5s
  TuyAPI {
  TuyAPI   devId: '23804861d8bfXXXXXXXX',
  TuyAPI   gwId: '23804861d8bfXXXXXXXX',
  TuyAPI   uid: '',
  TuyAPI   t: 1601557917,
  TuyAPI   dps: { '1': true }
  TuyAPI } +0ms
(node:9376) UnhandledPromiseRejectionWarning: Error: No connection has been made to the device.
    at TuyaDevice._send (/srv/openhab2-conf/scripts/test/tuya-mqtt/node_modules/tuyapi/index.js:235:13)
    at /srv/openhab2-conf/scripts/test/tuya-mqtt/node_modules/tuyapi/index.js:215:14
    at new Promise (<anonymous>)
    at TuyaDevice.set (/srv/openhab2-conf/scripts/test/tuya-mqtt/node_modules/tuyapi/index.js:212:12)
    at /srv/openhab2-conf/scripts/test/tuya-mqtt/tuya-device.js:158:25
    at new Promise (<anonymous>)
    at TuyaDevice.set (/srv/openhab2-conf/scripts/test/tuya-mqtt/tuya-device.js:157:16)
    at processDpsKeyCommand (/srv/openhab2-conf/scripts/test/tuya-mqtt/tuya-mqtt.js:314:16)
    at /srv/openhab2-conf/scripts/test/tuya-mqtt/tuya-mqtt.js:413:29
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:9376) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:9376) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Seems like the IP couldn't be detected. But this would be a normal behaviour in my environment. I seperated all IoT devices into it's own VLAN. Only the Raspberry (where I am running tuya-mqtt) is allowed to communicate with this VLAN and vice versa... But the Raspberry itself is in my Live VLAN (IP-Address 192.168.100.3/23) whilst the tuya-devices having 192.168.103.xxx/24.

Would it be possible to add the ip-addresses to the devices.conf manually and read them from there?

[tsightler]

Great, please poke around with it and let me know how it works. The core DPS logic is unchanged in the next beta, with the exception that the "custom" device will let you configure the list of DPS topics that should be queried for state during startup since right now there's no state information until you make a change.

I had previously planned to try to discover this automatically, a schema query works for a lot of older devices, but the new ones, they just ignore that or return null data via TuyAPI, so I'm going to have to table that idea for now. I don't think having to specify the DPS values is that huge of an issue for people that have already had to go through the effort to get device keys from Tuya!

[Flip76]

Just a thought... Would it be possible to add the dps-value in the devices.conf? So for example: One monitor is on the powerstrip dps:1 and the other one on dps:2.... If I can define the dps in the config, I would be able to use a friendly name there too... Something like:

  {
    name: 'Steckdosenleiste',
    id: '23804861d8bfXXXXXXXX',
    key: '868c7fa1XXXXXXXX',
    ip: '192.168.103.15',
    version: '3.3'
  },
  {
    name: 'Monitor 1',
    id: '23804861d8bfXXXXXXXX',
    key: '868c7fa1XXXXXXXX',
    ip: '192.168.103.15',
    dps: '1',
    version: '3.3'
  },
  {
    name: 'Monitor 2',
    id: '23804861d8bfXXXXXXXX',
    key: '868c7fa1XXXXXXXX',
    ip: '192.168.103.15',
    dps: '2',
    version: '3.3'
  }

[Flip76]

That's even better!! Great idea!! The big problem with home automation systems (like openhab) is the conversion for switches (on/off) to (true/false)... You always need to transform that values and in case of openhab a simple map-transformation doesn't seem to work. So I guess I have to write a little js to convert those values to switches and vers visa...

[Flip76]

Oh I see the on/off mqtt-channels in openhab offer a custom setting to handle this - now it works as expected :-)

[tsightler]

Yep, that was the idea, there are just so many devices out there there's no way I'm going to be able to support them all, as can be seen by other projects trying to do so. So, in the end, the idea was to create a tool that exposes all of the data in a raw way, and allows users to either use that (if they are advanced they can use the DPS/<#> or DPS JSON methods with whatever transform engine they are comfortable with) or, in other cases, just create a simple mapping using the custom device. Once someone creates a mapping that works, it's quite east to turn that into a "supported" device that can be referenced as a specific device type without having to map as the device files simply use the very same template engine, they are just codified into a device file so that you can reference them by name in the config instead of having to put the entire template there, and they include options to override specific variables, etc., similar to homebridge-tuya. Trying to strike a balance between "simple" and "flexible".

As it turns out, tuyagateway is using a similar concept, with a web based tool to configure, but I couldn't find a lot of people having success with it and, after playing with it for a couple of days, I never could get even my simple dimmer to do anything other than turn on/off (couldn't control brightness, etc).

I'm really close to meeting this goal, but I'm starting to run out of time to work on it. Hopefully I'll get it done tonight or by this weekend and tested enough to push beta2 and you can play around with it and let me know how it works for you, but it's good to know that to core code for 3.0 is working for you even now the next beta should be far more reliable.

[Flip76]

I absolutely agree, tuyagateway and gismocaster seems way to complicated for that. I will setup all my tuya devices and will have an play around with it over the weekend!

Thank you so much for this great tool!

[tsightler]

I've pushed 3.0.0-beta2 to the dev branch. There's still a lot of work to do, and there's only pre-built support for 3 specific devices (simple switch, dimmer, and RGBTW light, although only those with HSB style color, not HSBHEX, and no color temperature).

However, I've also implemented the generic device support so that you can build simple templates for devices like described above. Using the example above you'd just need:

{
  name: 'Steckdosenleiste',
  id: '23804861d8bfXXXXXXXX',
  key: '868c7fa1XXXXXXXX',
  ip: '192.168.103.15',
  version: '3.3',
  type: 'GenericDevice',
  template: {
    monitor1_state: {
      key: 1,
      type: 'bool'
    },
    monitor2_state: { 
      key: 2,
      type: 'bool'
    }
  }
}

Other types exist such as "int" and "str" and "hsb", you can see examples in the current device files as the device specific files are basically nothing but pre-defined templates that allow setting variables in the config file and a device specific discovery function for Home Assistant.

There's still some behaviors that I need to fix, for example, if a device gets an update it updates every attributes instead of just the one that changed. I don't like this, and it doesn't impact most use cases to get updates with the same data, but I plan to fix it.

For RGBTW lights I don't like how white mode support is implemented. Right now the bulb switches to color mode any time you set a color, and switch to white mode if you update the white level. I plan to also implement switch to white mode if saturation is zero or if you change color temperature (however, the one light I own doesn't happen to have color temperature, but I think I can implement it without real).

But, overall, I'm really happy where the base of this code is now. Yes, there's lots of improvements I could make, but it's been working very reliably with my devices for the last week.

If you do update, please make sure to run 'npm i' to update TuyaAPI to the custom version I'm now using which addresses some limitations in the official version.

[Flip76]

Thanks for all your effort! I pulled the new version, adjusted the devices.donf but for some power plugs I get an error (which I didn't see in the previous version):

 tuya-mqtt:mqtt Received MQTT message ->  {"topic":"tuya/tisch/command","message":"ON"} +2s
  mqttjs:client _sendPacket :: (mqttjs_4e406ada) ::  start +1ms
  mqttjs:client sendPacket :: packet: { cmd: 'puback', messageId: 30, reasonCode: 0 } +1ms
  mqttjs:client sendPacket :: emitting `packetsend` +0ms
  mqttjs:client sendPacket :: writing to stream +0ms
  mqtt-packet:writeToStream generate called +2s
  mqtt-packet:writeToStream generate: packet.cmd: puback +1ms
  mqtt-packet:writeToStream writeVarByteInt: writing to stream: <Buffer 02> +0ms
  mqtt-packet:writeToStream writeNumberCached: number: 30 +0ms
  mqtt-packet:writeToStream writeNumberCached: <Buffer 00 1e> +0ms
  mqttjs:client sendPacket :: writeToStream result true +2ms
  mqttjs:client sendPacket :: invoking cb +0ms
(node:22625) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'hasOwnProperty' of undefined
    at GenericDevice.processDeviceCommand (/srv/openhab2-conf/scripts/tuya-mqtt/devices/tuya-device.js:130:47)
    at GenericDevice.processCommand (/srv/openhab2-conf/scripts/tuya-mqtt/devices/tuya-device.js:184:18)
    at MqttClient.<anonymous> (/srv/openhab2-conf/scripts/tuya-mqtt/tuya-mqtt.js:133:32)
    at MqttClient.emit (events.js:315:20)
    at Arguments.<anonymous> (/srv/openhab2-conf/scripts/tuya-mqtt/node_modules/mqtt/lib/client.js:1259:27)
    at Object.MqttClient.options.customHandleAcks (/srv/openhab2-conf/scripts/tuya-mqtt/node_modules/mqtt/lib/client.js:183:150)
    at MqttClient._handlePublish (/srv/openhab2-conf/scripts/tuya-mqtt/node_modules/mqtt/lib/client.js:1252:15)
    at MqttClient._handlePacket (/srv/openhab2-conf/scripts/tuya-mqtt/node_modules/mqtt/lib/client.js:410:12)
    at work (/srv/openhab2-conf/scripts/tuya-mqtt/node_modules/mqtt/lib/client.js:321:12)
    at Writable.writable._write (/srv/openhab2-conf/scripts/tuya-mqtt/node_modules/mqtt/lib/client.js:335:5)
(node:22625) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 8)

[Flip76]

I was able to get rid of the errer when I use SimpleSwitch instead of GenericDevice

[tsightler]

Can you please post the device.conf file you used when you received the error?

…

On Sat, Oct 3, 2020 at 12:59 AM Flip ***@***.***> wrote: I was able to get rid of the errer when I use SimpleSwitch instead of GenericDevice — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#41 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEHV7G3I5EENS4UQNASZTA3SI2VUVANCNFSM4RSHYLBQ> .

[Flip76]

With this config I've got the error (but it was my fault case this device is just an ordinary power plug):

  {
    name: 'Tisch',
    id: 'bf0751f4b74d39XXXXXXXX',
    key: 'eacb979dXXXXXXXX',
    ip: '192.168.103.14',
    version: '3.3',
    type: 'GenericDevice'
  }

And with this config it works perfect:

  {
    name: 'Tisch',
    id: 'bf0751f4b74d39XXXXXXXX',
    key: 'eacb979dXXXXXXXX',
    ip: '192.168.103.14',
    version: '3.3',
    type: 'SimpleSwitch'
  }

[tsightler]

OK, so no custom template for the GenericDevice, that's the issue. By default (i.e. with no custom template) the GenericDevice doesn't create any friendly topics at all because it doesn't know anything about the device so it doesn't know what to create. Using the GenericDevice this way you can only access the device via the DPS topics.

However, the command processing code currently expects there to be at least one friendly topic. I admittedly didn't test the "no template" case very well (I fired it up and monitored for DPS state changed, but didn't send commands). It's easy to fix, I'll push an update shortly.

[tsightler]

I pushed a fix for this case to dev, just initialized deviceTopics to empty if there's no template, that should do it.

[Flip76]

Thanks! Will pull it in a minute. I am just setting up all tuya devices in order to add them to the devices list

[Flip76]

So I played around and for the switches / power strips everything works perfect! Next I tried to get the devices behind the Zigbee-Gateway to run but I couldn't make it...

Here is the output of "tuya-cli wizard":

[
  {
    name: 'Zigbee Wireless Gateway（TYGWZW-01）',
    id: 'bf1568c46aaec7XXXXXXXX',
    key: 'a1214b1fXXXXXXXX'
  },
  {
    name: 'Temperatur- und Feuchtigkeitssensor',
    id: 'bf86ac4f5ce854XXXXXXXX',
    key: 'a1214b1fXXXXXXXX'
  },
  {
    name: 'Security Remote Controller',
    id: 'bfd6d2a11bae7dXXXXXXXX',
    key: 'a1214b1fXXXXXXXX'
  }
]

As you can see, sub-devices of the gateway have different id's but they always have the same key. So I tried to different combinations in the devices.conf:

  {
    name: 'TempSensor',
    id: 'bf86ac4f5ce854XXXXXXXX',
    key: 'a1214b1fXXXXXXXX',
    ip: '192.168.103.12',
    type: 'GenericDevice',
        template: {
            temperature_state: {
              type: 'str'
            },
        }
  }

---

  {
    name: 'Gateway',
    id: 'bf1568c46aaec7XXXXXXXX',
    key: 'a1214b1fXXXXXXXX',
    ip: '192.168.103.12',
    type: 'GenericDevice',
        template: {
            temperature_state: {
              id: 'bf86ac4f5ce854XXXXXXXX',
              type: 'str'
            },
        }
  }

---

  {
    name: 'Gateway',
    id: 'bf1568c46aaec7XXXXXXXX',
    key: 'a1214b1fXXXXXXXX',
    ip: '192.168.103.12',
    type: 'GenericDevice',
        template: {
            temperature_state: {
              id: 'bf86ac4f5ce854XXXXXXXX',
              key: 'a1214b1fXXXXXXXX',
              type: 'str'
            },
        }
  }

As you see, in the 1st example I tried to add the temperature sensor as a normal device. In the 2nd example I tried to use the Gateway and the sensor as a Generic Device with id only. The 3rd try was same as the 2nd only adding the key too...

[Flip76]

Anyhow, I will go ahead trying the lightbulbs and I also found a Tuya universal IR Remote Control and a WiFi relais in my toybox :-)

And this is the temperature sensor I‘m trying to add:

[Flip76]

I succeeded in adding the power plug with meter:

{ 
    name: 'SwitchMeter',
    id: '2302731368c6XXXXXXXX',
    key: '79728e3cXXXXXXXX',
    ip: '192.168.103.26',
    version: '3.3',
    type: 'GenericDevice',
        template: {
            power_state: {
              key: '1',
              type: 'bool'
            },
            kwh_today_state: {
              key: '2',
              type: 'int'
            },
            current_ma_state: {
              key: '4',
              type: 'int'
            },
            current_w_state: {
              key: '5',
              type: 'int'
            },
            voltate_state: {
              key: '6',
              type: 'int'
            }
        }
  }

Now it get's even clearer what the key in the template means... It's the DPS right?

[tsightler]

Yes, that's correct, the key is the DPS key. It's sounds like you're having pretty good success so far.

I have no idea how the Tuya gateway works but it's not surprising to me that it doesn't work as I've never seen one. I see you can get the ids/keys from tuya-cli, so my question is, can you control it with tuya-cli? If you can, then it should be possible to get it to work. That being said, I'm pretty confident a better option would be to just get a Zigbee stick and control them directly.

[Flip76]

The funny thing is that I can use tuya-cli wizard only... All other commands (set/get) just fail with parse error for all my devices (even the switches which I can control with tuya-mqtt)...

On the other hand, the subdevices of the Zigbee-gateway are only sending data (not receiving) - like temp sensor and the remote control...

But you are right! I already looked at zigbee-mqtt as I have a couple of Xiaomi-devices which I want to control via zigbee-mqtt. I already ordered the stick a couple of days ago...

What I wanted to ask... In the mqtt explorer I see that another topic „homeassistant“ appeared, representing the state_topics from tuya-mqtt... Do you write to a second topic somewhere?

One other thing I realized... Above (at my last posting) you see that I configured some keys as int... But you initially send „None“ as value which causes an error on OpenHAB side as it expects a number... Would it be possible to send 0 to int-defined keys?

[tsightler]

First, let me say thanks to you for all the testing. Having what amounts to nearly a complete rewrite tested on so many devices out of the gate is hugely valuable, especially since I only have a small number of devices myself.

If you do test the light bulb, make sure to pull the latest code I pushed last night, lots of fixes there for the color support and also added support for old style HSBHEX bulbs, although you have to manually specify the type (I'll try to auto-detect in a future version). It still has a few issues, especially if it receives different commands for H, S, B separately in fast order. I need to add some code to deal with this rapid command issue.

The topics you are seeing being written to homeassistant are MQTT discovery messages. Home assistant implemented their own protocol for allowing devices to send a specially formatted MQTT message with all of the Home Assistant configuration data so that users don't have to do anything for devices to "just work". In home assistant my switch/socket/dimmer/light all just show up automatically. This of course doesn't work with the generic device because it has no way to now how the friendly topics should be mapped to Home Assistant settings. However, this might still be useful for OpenHAB as I believe it implements Home Assistant style auto topic discovery, although I'm not sure how complete the support is (in Home Assistant you can auto-discover any MQTT device with a supported integration).

Regarding the Zigbee gateway, just looking at the output from the wizard, I'm thinking that's the reason there is a gwId parameter for DP_QUERY. Right now the TuyAPI code always sets the DevID and GwID to the same, because, but I'm betting that with a gateway you should set the GwID to the DevID for the gateway while the DevID should be the actual DevID you want to talk to. Admittedly this is just conjecture on my part as I can find no code that talks the Tuya protocol that appears to support Zigbee gateways. I'm thinking it might be possible to figure it out if I had access to one, but the requests to support it aren't very common, probably because it's just easier to use Zigbee directly and bypass the Tuya stuff.

It's not an all surprising that you can use tuya-cli with "wizard" but not other command. The "wizard" command is not talking to the device, rather it's talking to the Tuya cloud app account to get the device keys. All other commands attempt to talk to the device. Lots of newer tuya devices do not respond to the TuyAPI get() request, so tuya-cli can set values, and get the response, but not get current values. As a matter of fact, the dev version of tuya-mqtt is currently using a modified TuyAPI that I branched which attempts to work around these issues by quietly issues a set() with a null value when the get() command fails, for all devices I have access to this manages to cajole them into sending a response. It seems this is mostly working for your devices as well.

Thanks for the feedback on the issue with int values. I certainly shouldn't provide a string in that case. However, I wonder if I should just suppress any non-int updates at all vs sending a zero, since that would not accurately reflect the value from the device. I guess I'm not even sure how it's managing to get some invalid value as, theoretically even the initial value should be valid. I guess I need to dig through that code to see if I can figure that out. Does the DPS value always reflect an integer?

I still need to implement "float" as well. I think with bool/int/float/string/hsb/hsbhex that should be enough to build a template for most devices, although there might be a few that need something special, covers for example.

[tsightler]

Regarding the int issue you reported, it's really good feedback. I wonder if you could test the latest code that I pushed late last night. That code implements granular DPS updates. With the prior code, any time data was received from the device it updated all topics, while now, data updates only trigger updates of the specific values/topics that changed. I'm hopeful that might get rid of the erroneous data. Regardless, I will probably try to implement some additional logic for this.

Once again, thanks very much for your valuable feedback and willingness to help test when the code is in such a highly fluid state. I hope that once you're done, you'll share your experience with the OpenHAB community.

[Flip76]

Oh no, you don‘t need to thank me! I have to thank you for the great work! Rest assured that I will let the OH-community know about this great way to handle tuya devices. There are many people looking for a reliable solution and yours is definitely the most promising!

Before I always looked into getting Homebridge devices exposed to mqtt but I didn‘t get it to work (but still would be nice to have). Admittedly, I am not a JS/Java/Python guy. My world is SQL,PL/SQL, Shell- and Perl-Scripting... But it seems I should definitely get a heads up to those techniques...

So if I understood right, you are creating the topics for Home Assistant? Indeed, I immediately saw those channels in the Openhab Inbox - so autodiscovery works fine! I will have a play around tomorrow with this too! Also I will pull the latest version and check for the int-issue...

Btw: If you are interested in having a play with my devices, I can setup a new tuya account, add my toys there and grant you access to my lab-env (which contains the latest stable OH-version 2.5.9) next to tuya-mqtt...

[Flip76]

PS: When you create a mqtt channel in OH, you have a list of choices:

Regarding to your question if the DPS always reflects an int-value... The int-value I tested was the voltage of the switch. This one should always be integer. But there might be other cases... Given that you can define the data type for each dps in the template maybe it would make sense to „initialize“ the value with a default of the data type. As to my knowledge, OH always accepts NULL for any data type. Another approach could be to let the user define the initialization value within the template?

[tsightler]

Yeah, I guess what I'm not understanding is how there could be a data update in my code that is somehow sending "None". I mean, I know what part of the code is generating this, but the check is there to cover the case where a user defines a field as Int, but the actual value returned by the DPS key is not an Int. I wanted an easy way to identify this case if someone did that. However, somehow this is happening in your case even though it seems obvious you have selected a DPS that can only be an int.

I've also been considering adding the ability to define some simple math transforms on DPS values when they are Int/Float values (and perhaps some regex support for str values). Do you think that would be useful? I know you can usually do this in the automation platform, but sometimes those tools can be quite unfriendly. I was thinking just simple add/subtract/multiply/divide functions so you could have "key" = 1, "type"="int" and "keyTranform"="*10" to, for example, multiple the int value by 10.

[tsightler]

One thing I had a question about, with the RBGTW light, I've been tweaking on that code and trying to figure out how to deal with switching between "white" and "colour" mode. Obviously there's a mode topic for manual switching, so that's fine, but I've found that a lot of Home Automation systems don't seem to have any specific switch for changing modes so, to make it work reasonably, you have to change modes automatically based on the color data provided.

This is the behavior currently:

Switch to color mode when receiving any hue (H) or brightness (B) change, or when receiving as saturation (S) change with S > 0.

Switch to white mode when receiving any white level (W) change, or when receiving a saturation (S) change to S= 0.

Implementing this logic with Home Assistant seems to work perfectly. If a change the color on the color wheel to anything other than the center, I get color mode, if I set it to center (i.e. saturation 0) the light switched to white mode (this makes sense to me as 0% saturation is white). If I change the brightness (part of HSB) it switches or stays in color mode. Changing "White Level" slider automatically changes to white mode as well, and, once I implement color temerpature, changing that would also switcht to white mode.

As well as this seems to work for Home Assistant, it doesn't work as well when those lights are presented to other devices, especially integrations that have more limited color implementations, for example that only support combined brightness or named colors. Some of these don't send anything at all for "white", instead just sending the "on" command, which is just weird to me, but I guess is perhaps some behavior from Hue bulbs, which apparently always go to white on power on?

So, on these devices I have no way to actually get the bulb to white mode. Even changing the brightness will flip the light to color mode based on the rules above (because those devices don't support white level or understand modes).

It a really difficult problem to overcome. I could ignore brightness for mode switch, and just have brightness control the brightness in color mode or white mode (just doing the DPS translation silently on the backend based on the mode), but this means that, for example, you can't have different brightness for white and color mode, and a lot of bulbs are much brighter in white mode than in color.

Another option would be to only have brightness control to actually report and control the white level brightness when in white mode, and the color level brightness when in color mode. This really isn't "that" difficult, but seems a little ugly. However, this would still leave me with no real way to get the light into white mode from devices that don't have a white level or saturation control.

Perhaps I'm getting too in the weeds on this and there's not that many scenarios that don't support full color modes now, but, for example, right now I can't get my Alexa to set the bulb to white mode via Home Assistant using any method because, when I tell Alexa to set it to white, it just sends an "on" command and no color. My only thought is to have an option to "set to white with "on" command", but the problem with that is that Home Assistant sends on commands with every state update, so I'd have to have logic to attempt to suppress mode changes for a few hundred ms, if it's received in case it's a spurious ON (I have no idea why Home Assistant sends "on" every time, to me that seems unnecessary).

Anyway, I'd love to hear thoughts on the best way to deal with it. Perhaps hearing how some other tools like OpenHAB deal with such devices would help.

[Flip76]

I've also been considering adding the ability to define some simple math transforms on DPS values when they are Int/Float values (and perhaps some regex support for str values). Do you think that would be useful? I know you can usually do this in the automation platform, but sometimes those tools can be quite unfriendly. I was thinking just simple add/subtract/multiply/divide functions so you could have "key" = 1, "type"="int" and "keyTranform"="*10" to, for example, multiple the int value by 10.

That would actually help a lot... For example: The switch with meter delivers a voltage value of 2246 - so you have to divide by 10... But then it's not an integer anymore...

[Flip76]

Regarding the lights, I just checked on my Philips Hue lights and Openhab uses 3 channels (color, brightness, color_temperature) sending 0 as temperature seems to activate white-mode... But I will check with the tuya light bulbs...

[Flip76]

I just tried to add my 1st tuya test bulb:

{
name: 'Test Bulb',
id: '56100002c44fXXXXXXXX',
key: '848c69c9XXXXXXXX',
ip: '192.168.103.23',
type: 'RGBTWLight'
}

The channel immediately appears in the homeassistant-topic (and also in the openhab-inbox) but I can't see the device under the tuya-topic:

The log shows:

2020-10-05T09:28:12.812Z tuya-mqtt:tuya Connected to device Test Bulb (192.168.103.23, 56100002c44fXXXXXXXX, 848c69c9XXXXXXXX)
2020-10-05T09:28:12.814Z tuya-mqtt:tuya Home Assistant config topic: homeassistant/light/56100002c44fXXXXXXXX/config
2020-10-05T09:28:12.815Z tuya-mqtt:tuya {
  name: 'Test Bulb',
  state_topic: 'tuya/test_bulb/state',
  command_topic: 'tuya/test_bulb/command',
  brightness_state_topic: 'tuya/test_bulb/brightness_state',
  brightness_command_topic: 'tuya/test_bulb/brightness_command',
  brightness_scale: 1000,
  hs_state_topic: 'tuya/test_bulb/hs_state',
  hs_command_topic: 'tuya/test_bulb/hs_command',
  white_value_state_topic: 'tuya/test_bulb/white_value_state',
  white_value_command_topic: 'tuya/test_bulb/white_value_command',
  white_value_scale: 1000,
  unique_id: '56100002c44fXXXXXXXX',
  device: {
    ids: [ '56100002c44fXXXXXXXX' ],
    name: 'Test Bulb',
    mf: 'Tuya',
    mdl: 'RGBTW Light'
  }
}
2020-10-05T09:28:12.815Z mqttjs:client publish :: message `{"name":"Test Bulb","state_topic":"tuya/test_bulb/state","command_topic":"tuya/test_bulb/command","brightness_state_topic":"tuya/test_bulb/brightness_state","brightness_comma
nd_topic":"tuya/test_bulb/brightness_command","brightness_scale":1000,"hs_state_topic":"tuya/test_bulb/hs_state","hs_command_topic":"tuya/test_bulb/hs_command","white_value_state_topic":"tuya/test_bulb/white_value_state","white_value
_command_topic":"tuya/test_bulb/white_value_command","white_value_scale":1000,"unique_id":"56100002c44fXXXXXXXX","device":{"ids":["56100002c44fXXXXXXXX"],"name":"Test Bulb","mf":"Tuya","mdl":"RGBTW Light"}}` to topic `homeassistant/l
ight/56100002c44fXXXXXXXX/config`

Does the device need some additional parameters in the devices.conf?

[tsightler]

Does the device need some additional parameters in the devices.conf?

It technically doesn't need them, but they might be required based on the device. Obviously I need to create documentation for the different devices to show what parameters are available for each, but for now the easiest way is to simply open the file for the device (in this case rgbtw-light.js) and typically around line 7 or so there will be a commented section for "Set device specific variables" that looks like this (example from rgbtw-light):

this.config.dpsPower = this.config.dpsPower ? this.config.dpsPower : 1
this.config.dpsMode = this.config.dpsMode ? this.config.dpsMode : 2
this.config.dpsWhiteValue = this.config.dpsWhiteValue ? this.config.dpsWhiteValue : 3
this.config.whiteValueScale = this.config.whiteValueScale ? this.config.whiteValueScale : 1000
this.config.dpsColorTemp = this.config.dpsColorTemp ? this.config.dpsColorTemp : 4
this.config.dpsColor = this.config.dpsColor ? this.config.dpsColor : 5
this.config.colorType = this.config.colorType ? this.config.colorType : 'hsbhex'

The word after "this.config" is the configurable variable, and the value on the far right is the default, so in the case above the custom options and their defaults are as follows:

dpsPower: 1 <- DPS for light power state
dpsMode: 2 <- DPS for white/color mode
dpsWhiteLevel: 3 <- DPS for white mode brightness
whiteValueScale: 1000 <- Scale of the white value setting
dpsColorTemp: 4 <- DPS for white color temperature (Warning: not currently implemented)
dpsColor: 5 <- DPS for HSB color values
colorType: 'hsbhex' <- The Tuya color format for this bulb (hsb/hsbhex)

So, for example, my generic Tuya bulb that I purchase recently (which does not have color temperature) uses this config:

{
    name: 'Office Lamp Color',
    id: 'eba86a1731xxxxxxxxxxx',
    key: '6ce9xxxxxxxxxxxx',
    type: 'RGBTWLight',
    dpsPower: 20,
    dpsMode: 21,
    dpsWhiteValue: 22,
    dpsColor: 24,
    whiteValueScale: 1000,
    colorType: 'hsb'
}

For right now the defaults are simply the defaults that tuya-mqtt has always used for these values so I assume they must work with most older devices. I believe that perhaps the default settings are common for older bulbs and the settings I use above are common for newer devices so I was thinking, in the future, I could add some auto-detection logic. For example I could query DPS 2 and DPS 21, and if either contain the words "white" or "colour", then I can be pretty sure it's a bulb, then I could query DPS 5 and DPS 24 and determine if it's HSB/HSBHEX based on format. But I'll admit I have no idea how consist the DPS values are between devices. Looking around it doesn't seem they are super consistent.

So, otherwise, the easiest thing to do is to start with it as a generic-device, then play around with it and monitor the DPS values that are returned, then configure them as appropriate.

[tsightler]

BTW, I noticed that OpenHAB requires a single topic with comma separated HSB values and also requires the scale for saturation and brightness to be 100. Right now the RGBTW light uses the "native" brightness scale for the color type, which for Tuya HSB is 1000 and for HSBHEX is 255. Home Assistant can deal with this because it puts the H and S components on one topic, while the B component is on a separate topic where you can define the specific scale.

Based on the fact that OpenHAB appears to require 100 scale, and Home Assistant can work with that scale, I think I'm just going to modify the code to use 100 scale for all MQTT values and do the translation to Tuya scale values on the backend.

[Flip76]

Thanks again for the detailed information! Don't know why but I can't get the light bulb to appear as tuya-topic (not even when I define it as generic device)

[Flip76]

BTW, I noticed that OpenHAB requires a single topic with comma separated HSB values and also requires the scale for saturation and brightness to be 100. Right now the RGBTW light uses the "native" brightness scale for the color type, which for Tuya HSB is 1000 and for HSBHEX is 255. Home Assistant can deal with this because it puts the H and S components on one topic, while the B component is on a separate topic where you can define the specific scale.

Based on the fact that OpenHAB appears to require 100 scale, and Home Assistant can work with that scale, I think I'm just going to modify the code to use 100 scale for all MQTT values and do the translation to Tuya scale values on the backend.

Yes I saw this too :-)

==> /var/log/openhab2/openhab.log <==
2020-10-05 21:08:21.034 [WARN ] [ab.binding.mqtt.generic.ChannelState] - Command '1000' not supported by type 'ColorValue': 1000 is not a valid string syntax

This is from the autodiscovered item homeassistant/light/56100002c44fXXXXXXXX/config

[tsightler]

Thanks again for the detailed information! Don't know why but I can't get the light bulb to appear as tuya-topic (not even when I define it as generic device)

Hmm, weird. You can always run in debug mode (DEBUG=tuya-mqtt:*) and see what's happening.

[tsightler]

Also, would be interested to know if you can control the device with tuya-cli. In general, if it doesn't work with tuya-cli it probably won't work with tuya-mqtt as we both use the same TuyAPI on the backend to do the actual communication with the devices. Admittedly, current dev branch is using a TuyAPI that I modified to work better with newer devices as, with current release version of TuyAPI /tuya-cli, the get() function doesn't return anything, you can only get data updates.

However, tuya-cli should be able to control any Tuya device, so, for example, sending true false to dps 1 or to dps 20, should still be able to turn the bulb on/off with tuya-cli to at least prove that there's some level of control for the device (assuming the device uses DPS 1/20, it could always use something else).

[Flip76]

2020-10-05T14:42:22.352Z tuya-mqtt:tuya Search for device id 56100002c44fXXXXXXXX
2020-10-05T14:42:22.363Z tuya-mqtt:tuya Found device id 56100002c44fXXXXXXXX
2020-10-05T14:42:22.430Z tuya-mqtt:tuya Connected to device Test Bulb (192.168.103.23, 56100002c44fXXXXXXXX, 848c69c9XXXXXXXX)

But no Data from device :-(

You are right, I am not able to set a state via tuya-cli - the command just hangs. Don't know how to check but could this bulbs running protocol version 3.2 (which is not supported buy tuyapi yet)?

[tsightler]

Also, make sure that you are using the very latest code (I pushed code until the wee hours this morning) and be sure to run "npm i" in the tuya-mqtt to make sure the custom TuyAPI is installed, otherwise there could be cases where it would hang as in the official TuyAPI there's no timeout for send() commands for example, so it will just wait forever.

That being said the GenericDevice shouldn't be sending any get()/set() commands by default, it should just be listening for data events from the device. I've never seen a device that didn't send STATUS command events for data updates, but I guess they could be out there.

[tsightler]

It's possible it's a 3.2 device, I really don't know. I guess it's based on how deep you want to go. If you are able to get packet traces of the Tuya app talking to the device locally, but you'd have to block the device that is running the app from being able to access the internet to force it to talk locally/directly to the device, otherwise it talks to Tuya cloud. I've heard rumors of devices that even the app can't control locally, only via Tuya cloud, but I've never seen one of those devices.

Other options would be trying it with a simple test app using an alternative Tuya protocol implementation like Tuyaface (tuyagateway seems unreliable, but my experience with Tuyaface itself is pretty good).

[Flip76]

I did a get now before I issued a set via tuya-cli and it worked:

[22:20:38] root@openhab:/srv/openhab2-conf/scripts/tuya-mqtt# tuya-cli get --ip 192.168.103.23 --id 56100002c44fXXXXXXXX --key 848c69c9XXXXXXXX --all --protocol-version 3.3
{
  devId: '56100002c44fXXXXXXXX',
  dps: {
    '20': true,
    '21': 'white',
    '22': 1000,
    '23': 1000,
    '24': '000003e803e8',
    '25': '020e0d0000e80383000003e803e8',
    '26': 0
  }
}
[22:20:49] root@openhab:/srv/openhab2-conf/scripts/tuya-mqtt# tuya-cli set --ip 192.168.103.23 --id 56100002c44fXXXXXXXX --key 848c69c9XXXXXXXX --protocol-version 3.3 --dps 20 --set false
Set succeeded.

[tsightler]

Ah, great, so really, your device is very similar to mine (except mine will not respond to a get schema), and yours appears to actually have color temperature (so hey, once I implement that you can test it too) so a device.conf similar to what I posted above should work. Just make sure you update to the very latest code I pushed this morning.

[Flip76]

I do a "git pull" and a "npm i" everytime I start testing :-) But the device still doesn't appear as tuya-topic :-(
And I just saw thet tuya-cli get/set is only working when tuya-mqtt is stopped - but I guess that's normal right?

[tsightler]

Yep, normal, most devices only allow a single local connection at once. It's not clear to me how the device could work with tuya-cli and not with tuya-mqtt, underneath both use the same API and perform effectively identical functions. Are you able to perform tuya-cli get of a specific DPS?

Probably the best thing to do is to run with DEBUG=TuyAPI and see if you see the raw data coming from the device. That would hopefully at least tell me if something is hanging in tuya-mqtt or in TuyAPI.

[Flip76]

I can get a single DPS with tuya-cli:

tuya-cli get --ip 192.168.103.23 --id 56100002c44fXXXXXXXX --key 848c69c9XXXXXXXX --protocol-version 3.3 --dps 20
true

If I start tuya-mqtt it seems it doesn't get any dps values:

2020-10-05T15:52:21.944Z TuyAPI Socket connected.
2020-10-05T15:52:21.947Z TuyAPI GET Payload:
2020-10-05T15:52:21.948Z TuyAPI {
  gwId: '56100002c44fXXXXXXXX',
  devId: '56100002c44fXXXXXXXX',
  t: '1601913142',
  dps: {},
  uid: '56100002c44fXXXXXXXX'
}

But when I trigger an event, I can see tuyapi receives data:

# Power-off device via tuya app
  TuyAPI Received data: 000055aa00000000000000080000006b00000000332e330000000000000005000000011a29e98a83edd93f3e494cd9fb809d2c89b396e1e4429fd89d209d7c8360b96caaef1ecfb581cfebfb9c25078d346dc7b4b535ebc569b544b72cb2d564b1d27e340cf1b9034bed0eaf8255c97c34624ccf4b90d00000aa55 +5s
  TuyAPI Parsed: +2ms
  TuyAPI {
  TuyAPI   payload: '3.3\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0005\u0000\u0000\u0000\u0001\u001a)銃��?>IL����,�����B�؝ �|�`�l��\u001eϵ�����%\u0007�4mǴ�5��i�D�,��d��~4\f�\u0003K�\u000e��U�|4bL',
  TuyAPI   leftover: false,
  TuyAPI   commandByte: 8,
  TuyAPI   sequenceN: 0
  TuyAPI } +0ms

# Power-on device via tuya app
  TuyAPI Received data: 000055aa00000000000000080000006b00000000332e330000000000000006000000011a29e98a83edd93f3e494cd9fb809d2c89b396e1e4429fd89d209d7c8360b96c6d631d4336f2e0cb2fe112ef795cea04eece147447798171e3ae98f5012d87289e179e74db28a7949aff39752661dde2647dc1700000aa55 +4s
  TuyAPI Parsed: +2ms
  TuyAPI {
  TuyAPI   payload: '3.3\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0006\u0000\u0000\u0000\u0001\u001a)銃��?>IL����,�����B�؝ �|�`�lmc\u001dC6���/�\u0012�y\\�\u0004��\u0014tGy�q㮘�\u0001-�(�\u0017�t�(����9u&a��',
  TuyAPI   leftover: false,
  TuyAPI   commandByte: 8,
  TuyAPI   sequenceN: 0
  TuyAPI } +0ms

[Flip76]

One more thing: When I trigger the power-on/power-off via the tuya app, I see in the debug log (when running : DEBUG=* node tuya-mqtt.js)

tuya-mqtt:tuya Data from device not encrypted: 33ILB lmcC6yyct9ua +4s

[tsightler]

OK, that's super userful as it tells me that TuyAPI is failing to decrypt the payload, specifically this:

payload: '3.3\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0006\u0000\u0000\u0000\u0001\u001a)銃��?>IL����,�����B�؝ �|�`�lmc\u001dC6���/�\u0012�y\\�\u0004��\u0014tGy�q㮘�\u0001-�(�\u0017�t�(����9u&a��',

That should be the decrypted response from the status message (commandByte: 8), which should be the JSON representing the state change or, worst case, a human readable string (if it's an error). The above looks like it's instead a failed decrypt. This also explains why you get the message "Data from device not encrypted". Inside tuya-mqtt there is a super simple parser which attempts to detect if the response from TuyAPI is JSON or not and outputs the "Data from device not encrypted".

So clearly the problem is that TuyAPI is returning invalid data to tuya-mqtt, data that can't be acted on, so it's just logging it (I admittedly should change the log messages, that's a holdover from the old tuya-mqtt code) and ignoring it.

Now we need to focus on why TuyAPI is returning bad data when obviously it CAN return good data since it works with tuya-cli. So, I'm going to ask you to do something obvious, double/triple check that the key in devices.conf is correct. I know that's probably not the problem, because obviously you know what you're doing, but that's by far the most common reason for invalid decrypt, so I have to ask.

Assuming the key is correct, I'm wondering if you'd be willing to share your devices.conf file with me, with key in tact (via private email of course). It could be some type of parsing error on the key on my side, but if I had the key and can try parsing the "received data" packets in your debug output above and hopefully figure out this mystery.

[Flip76]

As I copy & paste the id, key and ip from the devices conf to the tuya-cli command, I'm pretty sure the values are correct - and I just double checked.... But..... Believe it or not: Just right now the device appeard in the tuya topic:

Honestly, I have no idea why it works now... The only thing I did was to change the friendly name from "Test Bulb" to "TestBulb" and stop/start tuya-mqtt again (like I did 20 times before)...

PS: Of course I can send you my devices.conf by mail if you want

[Flip76]

I went ahead and I checked the possibilities on OH (by crosschecking with my Philips Hue devices). For the hue-devices we typically use 4 channels:

• switch for on/off
• colorpicker (for color)
• dimmer (for brightness)
• dimmer (for color temp)

The mqtt-channel in OH gives me the following choices:

[tsightler]

Hmm...interesting, I guess it would be good to know what happens if you put the space back in, if that was the only thing that changed it could be some type of weird parsing issue, although I mostly just rely on the JSON5 package for this and all of my device friendly names have spaces in them. Anyway glad it's working now, at least proves it's nothing major from a code perspective.

Regarding color, how do you set the light to white mode if it's currently a color? Is it the color temp setting? How about the dimmer? Does it behave differently if the light is in white mode vs color mode? In other words, if the light is white does Dimmer set white brightness and if the light is in Color mode does it set the Color brightness? The Tuya bulb (and some others) have separate brightness for color mode vs white mode. This is easily handled in Home Assistant since it has both "brightness", which sets color mode brightness, and "White Level" which sets white mode brightness, but I'm just not sure if OpenHAB or others have such a concept.

[Flip76]

I put the space back in and it still works... So I can only guess that it might not have been a space there before (maybe a special character caused by file-encoding with my editor)...

Regarding the color mode: Seems like OH doesn‘t work the same way like HA.

I researched the forum and I found this (basically about Philips Hue but I think same applies for the Tuya light bulbs):

https://community.openhab.org/t/hue-hsb-how-to-find-correct-color-warm-cold-white/82964/7

I am just thinking of using a dimmer item to handle the white color and a switch to toggle between white/color mode - can‘t think of another workaround...

[Flip76]

Another possible approach is described here:

https://www.openhab.org/addons/bindings/milight/

Basically they switch to white mode if the saturation is below 50%...

[tsightler]

Another possible approach is described here:

https://www.openhab.org/addons/bindings/milight/

Basically they switch to white mode if the saturation is below 50%...

Yeah, this is basically what the code does now, switches to white mode if saturation is 0%, because 0% saturation is white! But, I could change it to say 10% The 50% value seems too high to me as there's quite a bit of difference between 10% of 50%, but I can barely perceive the difference between 10% and 0% so perhaps that's a good threshold.

The only problem with this is that it doesn't address the issue for visualizations that don't have such flexibility. For example, Alexa seems to only show on/off, brightness and a preset selection of colors. I can seem to set the saturation via Alexa so there's no way to get it back to white. When I tell Alexa to set the color to white, it just sends "on", which is just weird to me, although maybe I could build a special mode that, if it received "on", but no other commands within 200ms, it assumes a switch to white mode.

Oh well, I have ideas on how to deal with this, just some minor tweaks to the logic that's already there and I think it will be mostly good.

So what's left before I can publish at least a working version 3.0? I think the following:

1. Update RGBTW white/color logic
2. Add color temp support for RGBTW
3. Rebase brightness/white level in RGBTW to 100 instead of 1000 (works for both Home Assistant and OpenHAB).
4. Write some auto-discovery rules for RGBTW for the most common values
5. Add the basic math transform functions
6. Create some documentation

My goal is to do 1-5 tonight, and then work on 6 through the week, to hopefully publish 3.0 next weekend. Then I can start working on adding additional devices as users submit working templates (or some I can just figure out based on existing devices in tuya-homebridge).

It seems like, even in it's current state, the code is mostly working for you and your devices (minus the Zigbee gateway), is that a fair statement?

[Flip76]

Have you ever tried using „cold white“ with Alexa? At least this works for my Hue bulbs...

Tuya-mqtt 3.0 already exceeded my expectations! I was almost about giving up on controlling my tuya devices via OpenHab. Thanks to you everything works like a charme now!

3 more ideas (but I think the first 2 are mainly a job for tuyapi):

• Automatically identifying the protocol version (this might be very helpful - especially for beginners)

• Handling of non-reachable devices (for example in the case of power loss). Currently mqtt doesn‘t change the state to off when the device has been toggled by Alexa/Tuya app. I guess thats the nature of those devices as they can‘t send a last will right? But as the devices get handled locally, why not set the state to off/false when the device isn‘t reachable within the network? Maybe there is a good reason not to do so but I can‘t see it at the moment...

• Automatically reread the devices.conf on changes. This might be helpful when running tuya-mqtt as a systemctl service. So there is no need to restart the service after a change in devices.conf. Maybe it‘s worth to add a quick „how-to run tuya-mqtt as service“ to the documentation - would be helpful for beginners

But all those points are just goodies! tuya-mqtt does already more than anyone would expect!

I also monitored the cpu/memory usage throughout the whole week and it‘s working absolutely reliably without any memory/cpu leak.

About the zigbee-gateway: As you said it might be not too much work to implement it. If you wanna try it, I can always get you access to my lab-env. But as discussed before, the best solution might be to directly communicate via a usb-stick with the devices (still waiting for my stick to give it a try).

If you have a paypal-me, it would be a pleasure to get you a „big coffee“ :-)

[Flip76]

PS: Whenever you feel ready to release 3.0, please let me know and I will write a review for the OH-community.

[tsightler]

Have you ever tried using „cold white“ with Alexa? At least this works for my Hue bulbs...

Interesting, I'll try it. Maybe it will work once I implement color temperature.

• Automatically identifying the protocol version (this might be very helpful - especially for beginners)

Automatic protocol detection already works if you use discovery vs specifying the IP address. You just can't use it because if your network setup. It works by listening for UDP packets on the network the device is on, but that only works if the server running tuya-mqtt and the devices are on the same network. I suppose it would be possible for TuyAPI to simply try each protocol in succession if the initial connection doesn't work, but indeed that's mostly a function of TuyAPI (although theoretically I could probably implement it on the tuya-mqtt side as well).

• Handling of non-reachable devices (for example in the case of power loss). Currently mqtt doesn‘t change the state to off when the device has been toggled by Alexa/Tuya app. I guess thats the nature of those devices as they can‘t send a last will right? But as the devices get handled locally, why not set the state to off/false when the device isn‘t reachable within the network? Maybe there is a good reason not to do so but I can‘t see it at the moment...

I'm not 100% sure what the request means. If you change the state to off when it's toggled by Alexa/Tuya app, it should definitely reflect in the UI. Certainly for me every update I make in the app is properly reflected in Home Assistant nearly instantly. Perhaps you mean if the Tuya device itself is powered off completely or something? I actually did plan to implement availability status, which in Home Assistant properly shows the devices as available/unavailable, it will be pretty easy.

• Automatically reread the devices.conf on changes. This might be helpful when running tuya-mqtt as a systemctl service. So there is no need to restart the service after a change in devices.conf. Maybe it‘s worth to add a quick „how-to run tuya-mqtt as service“ to the documentation - would be helpful for beginners

Yeah, I'll probably just copy the instructions from ring-mqtt for the most part, just update for tuya-mqtt, and I'll almost certainly create a Docker build as well.

About the zigbee-gateway: As you said it might be not too much work to implement it. If you wanna try it, I can always get you access to my lab-env. But as discussed before, the best solution might be to directly communicate via a usb-stick with the devices (still waiting for my stick to give it a try).

The biggest problem is that it would require changes to TuyAPI, which is not really my project and will require a little more digging than I have time for right now. Perhaps I will try to find a hub on ebay or something and play around with it in the future.

If you have a paypal-me, it would be a pleasure to get you a „big coffee“ :-)

It's really appreciated but I always tell people to donate to some worthy cause instead. I'm in the fortunate place to not really need much, but there are plenty of people who could use a little help these days. If you did send me an amount I'd just donate it myself, so you can cut out the middle-man.

[Flip76]

Sorry for the lack in response - I had 2 busy days at work.

• Automatically identifying the protocol version (this might be very helpful - especially for beginners)

Automatic protocol detection already works if you use discovery vs specifying the IP address. You just can't use it because if your network setup. It works by listening for UDP packets on the network the device is on, but that only works if the server running tuya-mqtt and the devices are on the same network. I suppose it would be possible for TuyAPI to simply try each protocol in succession if the initial connection doesn't work, but indeed that's mostly a function of TuyAPI (although theoretically I could probably implement it on the tuya-mqtt side as well).

You are right - I forgot about my VLAN-situation here :-)

• Handling of non-reachable devices (for example in the case of power loss). Currently mqtt doesn‘t change the state to off when the device has been toggled by Alexa/Tuya app. I guess thats the nature of those devices as they can‘t send a last will right? But as the devices get handled locally, why not set the state to off/false when the device isn‘t reachable within the network? Maybe there is a good reason not to do so but I can‘t see it at the moment...

I'm not 100% sure what the request means. If you change the state to off when it's toggled by Alexa/Tuya app, it should definitely reflect in the UI. Certainly for me every update I make in the app is properly reflected in Home Assistant nearly instantly. Perhaps you mean if the Tuya device itself is powered off completely or something? I actually did plan to implement availability status, which in Home Assistant properly shows the devices as available/unavailable, it will be pretty easy.

Sorry for the misleading information... You are right - when I use Alexa/Tuya app, the update comes instantly. But if, for expample, somebody unplugs the device from power, the last state will remain until the device will be plugged again right?

About the zigbee-gateway: As you said it might be not too much work to implement it. If you wanna try it, I can always get you access to my lab-env. But as discussed before, the best solution might be to directly communicate via a usb-stick with the devices (still waiting for my stick to give it a try).

The biggest problem is that it would require changes to TuyAPI, which is not really my project and will require a little more digging than I have time for right now. Perhaps I will try to find a hub on ebay or something and play around with it in the future.

Never mind! I'm looking forward to have a play with zigbee2mqtt once my usb-stick arrives

If you have a paypal-me, it would be a pleasure to get you a „big coffee“ :-)

It's really appreciated but I always tell people to donate to some worthy cause instead. I'm in the fortunate place to not really need much, but there are plenty of people who could use a little help these days. If you did send me an amount I'd just donate it myself, so you can cut out the middle-man.

Fair enough! I just contributed to a nearby Orphanage as here in TH they are not supported by the state and the Covid-situation hit them hard too!

[tsightler]

Sorry for the lack in response - I had 2 busy days at work.

Ha, no issues, I'm also quite busy these days. I've managed to get most of the work above done, but still need to implement color temperature controls for lights (I've ordered a light that actual has color temp to make this easier for me to test).

I think I found and fixed the hanging problem you mentioned above, it was really an error in TuyAPI that could cause set() to hang forever, especially if you didn't use await as concurrent set requests could reproduce a hang easily. This was made wrose because I had also implement fallback from DP_QUERY to sending a CONTROL with null for newer devices that don't respond to DP_QUERY, but this meant that set() was being called even for get() and had more potential to hang. I've implemented a queue and timeout for set() in my branch of TuyAPI (and submitted a PR to the original project).

I've also implement automatic discovery for RGBTW lights, it attempts to determine if the light is old/new style by querying DPS 2 and then DPS 21 and then checking the color value for HSBHEX vs HSB format. This should make most RGBTW lights work without needed any config, but of course the config options are still there..

I've also rebased all values in RGBTW lights to 100, instead of previously where they were the raw scale used by the Tuya bulb itself (255 or 1000) and I added a hsb_state topic that includes all three values in comma separated format as that's what it appears OpenHAB uses. This was easy as I built the HSB/HSBHEX types to use a components template. I'd be interested to know if that works easier with OpenHAB now.

Finally, I added some simple math expression evaluation (right now using eval but I will probably change it to use MathJS evaluate function). Use is pretty simple I think, here's a simple example of converting 1000 scale to 100:

brightness_state: { 
                key: 2,
                type: 'int',
                min: 1,
                max: 100,
                scale: 100,
                stateMath:  '/10',
                commandMath: '*10'
            }

Basically "stateMath" is the math to perform on the source value before being published as the state value, and "commandMath" is the math to be performed on the received command before being sent to the device. The min/max values are validate prior to command math. The above example takes the 1000 scale "white value" from my dimmer and translates it to 100 scale, by applying "stateMath" transform, and then, when I send values to the command topic it validates that the command is between the value of 1-100 and then performs "commandMath" to translate the value back to the 1000 scale.

For int values it performs the math and then, rounds and converts back to integer. For float values you get the decimal values. Thoughts on this feature are welcome as I wasn't sure of the best way to do it, but the above seemed pretty easy overall.

Fair enough! I just contributed to a nearby Orphanage as here in TH they are not supported by the state and the Covid-situation hit them hard too!

Thanks for that! Exactly the type of place that I would have looked for.

[tsightler]

Thanks for the testing, I've really put a lot of effort into getting the logic down and it really seems to work well for me, but it's always good to have validation so I'm glad to hear it's working well for you.

I've actually already changed the naming at this point, although hopefully, with Home Assistant, the MQTT discovery just works so no real need to know about them, but I thought it might be a little more clear for others, and especially for non-HA users.

Regarding the issue with Home Assistant showing the bulb icon in color, the issue here is that Home Assistant doesn't really have any concept of white/color mode. Yes, I have a topic for it, but it's mostly for the non-Home Assistant use case. Basically, Home Assistant itself is completely ignorant that white/color mode exist. Because of this, Home Assistant always shows an approximation of the bulbs current HSB value but, when the bulb is in white mode, the HSB value does matter for the Tuya bulb as it only uses white level/color temp in that mode. So basically, the bulb is in white mode, but Home Assistant still uses the color value.

The way around this is if you set the saturation to zero, which is also a way to switch to white mode, then Home Assistant knows that there is no color and the icon changes to the normal color. I've considered automatically setting the saturation to zero whenever white mode is enabled, which would work around this limitation with Home Assistant, but this has a side effect I don't really like. Right now, assume you have to light set a color like blue, with full saturation and brightness. If you then do nothing but switch from color to white mode, and then immediately back to color mode (for example using the mode topic or the app) the light will still have the exact same color settings as before the switch, so it will still be blue with full saturation and brightness.

However, if I switched the behavior to set the saturation to zero automatically whenever white mode is activated then performing the simple step of switch from color->white->color would lead you with a bulb in color mode but with 0% saturation. That just seems wrong to me.

I do have an idea how I can use an already existing feature of the code to work around this problem with minimal changes. I'll give it a shot and see how it works.

[tsightler]

After thinking about the "Home Assistant shows color even when light is in white mode" issue for a night I came upon a solution that seemed super simple which I've implemented in a current code push. The logic is as follows:

On any switch between color/white mode, no matter how that switch is triggered, it will always trigger a resend of the HSB color states to the friendly device topics. If the light is in color mode, the lights actual HSB value is sent, but, if the light is in white mode, the H and B values are sent as normal, but saturation value is always sent as 0%. Since it's only overriding the state in the friendly topics, not actually changing the bulbs state, if you change modes in, for example, the Tuya/SmartLife app between white/color, Home Assistant will always show the correct state regardless. I also think this more accurately reflects the bulb state since white mode is effectively 0% saturation.

@Trauma If you have a chance to test these latest minor changes, I'd really appreciate your feedback.

This leaves me with just a few minor things left, I need to monitor the Home Assistant status topic so I can send state updates whenever Home Assistant is restarted, and I need to write all of the documentation. Then I can publish 3.0 and start working on expanding device support to more devices.

[Flip76]

A short update from my side: Everything works so far but I can see a lot of connection timeouts (almost every 1 minute):

Oct 14 23:39:57 openhab systemd[1]: Started tuya-mqtt.
Oct 14 23:40:17 openhab node[32087]: /etc/openhab2/scripts/tuya-mqtt/node_modules/tuyapi/index.js:337
Oct 14 23:40:17 openhab node[32087]:           reject(new Error('connection timed out'));
Oct 14 23:40:17 openhab node[32087]:                  ^
Oct 14 23:40:17 openhab node[32087]: Error: connection timed out
Oct 14 23:40:17 openhab node[32087]:     at Socket.<anonymous> (/etc/openhab2/scripts/tuya-mqtt/node_modules/tuyapi/index.js:337:18)
Oct 14 23:40:17 openhab node[32087]:     at Object.onceWrapper (events.js:420:28)
Oct 14 23:40:17 openhab node[32087]:     at Socket.emit (events.js:314:20)
Oct 14 23:40:17 openhab node[32087]:     at Socket._onTimeout (net.js:484:8)
Oct 14 23:40:17 openhab node[32087]:     at listOnTimeout (internal/timers.js:554:17)
Oct 14 23:40:17 openhab node[32087]:     at processTimers (internal/timers.js:497:7)
Oct 14 23:40:17 openhab systemd[1]: tuya-mqtt.service: Main process exited, code=exited, status=1/FAILURE
Oct 14 23:40:17 openhab systemd[1]: tuya-mqtt.service: Failed with result 'exit-code'.
Oct 14 23:40:17 openhab systemd[1]: tuya-mqtt.service: Service RestartSec=100ms expired, scheduling restart.
Oct 14 23:40:17 openhab systemd[1]: tuya-mqtt.service: Scheduled restart job, restart counter is at 385.
Oct 14 23:40:17 openhab systemd[1]: Stopped tuya-mqtt.

I am running tuya-mqtt as system service:

[Unit]
Description=tuya-mqtt
After=mosquitto.service

[Service]
#ExecStart=/usr/bin/node /etc/openhab2/scripts/tuya-mqtt/tuya-mqtt.js
ExecStart=/usr/bin/node --unhandled-rejections=strict /etc/openhab2/scripts/tuya-mqtt/tuya-mqtt.js
#ExecStartPost=/etc/openhab2/scripts/tuya-mqtt/initialize_topics.sh
Restart=always
User=openhabian
Group=openhabian
Environment=PATH=/usr/bin/
Environment=NODE_ENV=production
WorkingDirectory=/etc/openhab2/scripts/tuya-mqtt/

[Install]
WantedBy=multi-user.target

As you can see, before I started with:

ExecStart=/usr/bin/node /etc/openhab2/scripts/tuya-mqtt/tuya-mqtt.js

This caused the service to stop without a restart... Now I am running with the "--unhandled-rejections=strict"-option and at least the service will automatically be restarted. I don't really know why I see so many connection lost messages... Any chance that the timeout is set to low?

PS: I will pull the latest version over the weekend and will run the tests again with the tuya-lightbulbs and OH...

[tsightler]

Maintaining the connection is mostly the responsibility of TuyAPI, however, early betas had no real handling for lost connections. The code from this past weekend should have much improved behavior as it now monitors the heartbeats from the device (every 10 seconds) and forces a disconnect and reconnect cycle to start if it misses 3 heartbeats in a row (it should also update the status topic to offline in this case so that it's clear the device is currently not reachable).

During the reconnect cycle it will try every 60 seconds to find and reconnect to the device. I'd tested this pretty extensively by removing power from my devices, blocking them from the network, etc. The expectation is that there should be no crashes!

One challenge with the local connection is that most devices will only accept a single connection at a time. If you use the Tuya app for local control it will kick the script out. Also, if anything causes a timeout, like a simple wifi issue or something, the device will not always realize that the old connection has been reset, I've seen it take 3-5 minutes before the device allows a new connection. Some devices also seem to take a long time after they first startup (if, for example, you remove power) before they allow the local connections to work, this is even reproducible with the Tuya app so I don't think there's anything I can do about it other than keep retrying.

[tsightler]

Another issue I discovered is that some devices practically stop responding if you send them some value outside of their expected range, but it's not always obvious that the value is not valid. For example, I have several lights/dimmers that use a 255 scale for the brightness. Seems reasonable to expect that you can send any value between 0-255, or at least 1-255, right? Well, it mostly works, but for a couple of the bulbs, any value <25 causes the light to start timing out and being very slow to respond to any commands, and if I send it 0, they turns off completely. To make it behave like the Tuya app I have to limit the range of values sent to 25-255.

There was also pretty bad bug in versions prior to beta4 that caused values 0-1 to sometimes be sent incorrectly as a true/false bool so it could be something like that causing the timeouts. I think I've squashed all of those bugs at this point.

I'm currently testing with a fairly wide arrange of switches, lights, dimmers, and various other devices with no real issues and I've seen no timeouts in the logs during the last week.

[Flip76]

Thank you so much! You gave me the "missing link".... I forgot about my homebridge-installation... I've used a tuya-binding there ( https://github.com/milo526/homebridge-tuya-web ) to expose the devices to Apple homekit... But with tuya-mqtt I don't need that anymore... I just removed the homebridge-plugin and exposed the mqtt channels from OH to homekit - which works like a charme :-)

[Flip76]

So I can finally confirm that after removing the tuya-plugin from homebridge there was no further disconnect - absolutely stable!

[tsightler]

Thanks a lot @Flip76. It's been extremely stable for me as well over the last week. I feel like it is really close to release quality. I was chasing a couple of corner case bugs in the color/white switching logic last night, but I think I finally have that nailed, and was even able to simplify the overall code in the process. I need to implement some type of scaling limit for the backend code to address the issue I mentioned above where a device doesn't accept commands for the full range (I've been looking at how some other projects address this and it looks simple enough). Now time to write some docs.

I really can't think you all enough for the testing, really helped me understand the use cases and it was invaluable to have feedback outside of my own testing and devices. There are still little things I'd like to do, but I think the core functionality is quite solid at this point.

[Flip76]

I just dropped you a PM on the OH community platform as here it doesn't seem to be possible :-)

[Trauma]

@Trauma If you have a chance to test these latest minor changes, I'd really appreciate your feedback.

I've just tested, it does the trick just fine, well done.

I have timeout connections too, not sure if it's a discovery problem or real connections issues. I do not use ip's in the device.conf, so problem might be related to discovery feature. In fact it works fine for 10mins approx then timeouts happens. 3.0.0-beta4 was perfectly stable, and my setup did not changed since my previous tests, I've only rebuild my docker image this morning with latest commit "Simplify color logic".

2020-10-16T21:01:19.382Z tuya-mqtt:error find() timed out. Is the device powered on and the ID or IP correct?
2020-10-16T21:01:19.382Z tuya-mqtt:error Will attempt to find device again in 60 seconds

[tsightler]

It's a little confusing because there have been no changes to this part of the code since 3.0.0-beta4 and also, you state that it works for 10 minutes, which means the device is already found at least once and the initial connection is established. Once the initial connection is established the only way for this message to happen again is if the device disconnects, either due to the device closing the connection, or missing 3 heartbeats in a row (basically, the device is not responding to heatbeat packets). Thus there should be more details in the logs further up to show this.

The find() function is really outside of this script, as it is part of TuyAPI but it's not really clear how this could possibly go wrong as it simply relies on finding the device via the UDP packets that they all send every 10 seconds. The only way for this to happen is if the device is not sending those packets which implies that it really is off the network.

[tsightler]

Squashed all the bugs I could find this weekend and finally managed to finish the documentation well enough to push 3.0.0 this weekend. Let's hope nothing bad snuck in with the last little changes but it seems really solid to me.

[Flip76]

I pulled 3.0.0 and will keep an eye on it :-)

[Flip76]

Will you push it to the master-branch soon?

[tsightler]

Will you push it to the master-branch soon?

It's done! Master branch is 3.0.0 as of ~12 hours ago. I still think the docs could use some work, but I'm not good at documentation.

[Flip76]

I'm blind... When I looked 8 hours ago I didn't see the master branch updated.... Maybe some Chrome-Cache issue :-( Now I see it! :-)

[Trauma]

It looks like gismocaster just stores it's config information in MQTT via retain, which I'm not a big fan of doing as I find people have a tendency to treat their MQTT broker as throwaway, they'll remove/reinstall it, etc.

As far as I can say tuya-mqtt v3 do not use retained messages at all ? From my perspective, i'dd prefer messages to be retained so devices availability persist across home assistant reboot. So actually I must restart tuya-mqtt after each HA restart, not convenient to me.

Don't you think HA discovery and status message should be retained ? Maybe it could be switched on/off through config file so everyone is happy.

EDIT : While looking at the code I've found that if "homeassistant/status" status message is resend every 30 sec. Which basically fits my request. Gonna try this.

[tsightler]

No retained messages and my goal is to always design solutions without any dependency on retained messages. I tried using retained messages for things in the early days of ring-mqtt and it was just a frustrating experiences of cases being opened because people couldn't delete devices (since config was retained in MQTT) or people deleting/reinstalling MQTT causing issues, or, in some cases, people not having any persistent storage configured for their MQTT broker (admittedly, I believe the last one is far less common these days).

However, the code as written does monitor for birth messages from Home Assistant which should be issued by default on Home Assistant 0.113 and newer (was finally made default exactly for this purpose). If it sees homeassistant/status message of "online" it knows that Home Assistant was restarted and, 30 second later, it will resend discovery and state data. This should require no special configuration in Home Assistant these days since birth messages are enabled by default. The code also monitors hass/status as well, since that was a common configuration for Home Assistant birth messages prior to 0.113. I have used this method since the earliest days of ring-mqtt and the ring-mqtt HomeAssistant addon and it's proven to be very functional, however, although it seems to work for me, I haven't fully tested the birth message implementation in tuya-mqtt as I didn't expect tuya-mqtt to be used often with Home Assistant since localtuya custom component provides similar functionality and is significantly easier to configure for most Home Assistant users.

[Trauma]

So I've made tests, and yes the whole behavior you describe happens, except for status message. Discovery, state and dps topics get populated just fine 30 sec after homeassistant/status turn to online, but status don't (I've deleted every topics before testing).