mhthies/smarthomeconnect

Python 3 AsyncIO-based home automation and interfacing framework

Smart Home Connect

Smart Home Connect (SHC) is yet another Python home automation framework—in line with Home Assistant, SmartHome.py, SmartHomeNG and probably many more. Its purpose is to connect "smart home" devices via different communication protocols, provide means for creating automation rules/scripts and a web interface for controling the devices via browser.

In contrast to most other home automation frameworks, SHC is completely based on Python's asynchronous coroutines (asyncio) and configured via pure Python scripts instead of YAML files or a fancy web engineering tool. Its configuration is based on instantiating Connectable objects (like state variables, User Interface Buttons, KNX Group Addresses, etc.) and interconnecting them with simple Read/Subscribe patterns. Thus, it is quite simple but really powerful, allowing on-the-fly type conversion, expressions for calculating derived values, handling stateless events, …. Read more about SHC's base concepts in the documentation.

Features

• interfaces
  • KNX bus via KNXD
  • DMX (via Enttec DMX USB Pro and compatible interfaces)
  • HTTP/REST API + websocket API
  • SHC client (connecting to another SHC instance via websocket API)
  • MIDI (esp. for Hardware MIDI controllers)
  • MQTT
  • Tasmota (currently: relais, RGB+CCW lights, IR receiver, power sensors; more features will be added on demand)
  • Pulseaudio
  • Telegram Bot
• websocket-based web user interface (using aiohttp, Jinja2 and Semantic UI)
  • widgets: buttons, text display, text/number inputs, dropdowns, images with placeable buttons, charts, etc., …
• configuration of data points/variables and automation rules in plain Python
  • full power of Python + intuitive representation and interconnection of different interfaces
  • type checking and extensible type conversion system
  • connecting objects via Python expressions
• chronological and periodic timers for triggering rules
• Logging/Persistence (no really stable in API yet)
  • to MySQL

Roadmap

• Stabilize logging API
• Logging to Influx-DB
• More web widgets
  • Gauges
  • timeline/"stripe" charts

Getting started

0. (Optional) Create a virtual environment to keep your Python package repositories clean:

   python3 -m virtualenv -p python3 venv
   . venv/bin/activate

   Read more about virtual environments in the offical Python docs.

1. Install the smarthomeconnect Python distribution from PyPI:

   pip3 install smarthomeconnect

   It will be only install smarthomeconnect and the dependencies of its core features. Additional depdencies are required for certain interface modules and can be installed via pip's/setuptool's 'extras' feature. See Depdencies section of this readme for a complete list. If you install SHC from a source distribution (in contrast to a "binary" package, such as a "wheel" package from PyPI), you'll need NodeJS and npm installed on your machine, which are used to download the web UI assets during the Python package building process.

2. Create a Python script (let's call it my_home_automation.py) which imports and starts Smart Home Connect:

   #!/usr/bin/env python3
   import shc
   
   # TODO add interfaces and Variables
   
   shc.main()

   When running this script (python3 my_home_authomation.py), SHC should start up and run idle until you terminate it (e.g. with Ctrl+C). See the code below for an example with the Web UI and the KNX interface.

3. Read about the basic concepts of SHC and available interfaces in the SHC documentation. Extend your script to create interfaces and Variables, connect connectable objects, define logic handlers and let them be triggered.

Simple Usage Example

import datetime
import shc
import shc.web
import shc.web.widgets
import shc.interfaces.knx

# Configure interfaces
knx_connection = shc.interfaces.knx.KNXConnector()
web_interface = shc.web.WebServer("localhost", 8080, "index")

web_index_page = web_interface.page('index')


# Simple On/Off Variable, connected to KNX Group Address (initialized per Group Read telegram),
# with a switch widget in the web user interface
ceiling_lights = shc.Variable(bool, "ceiling lights")\
    .connect(knx_connection.group(shc.interfaces.knx.KNXGAD(1, 2, 3), dpt="1", init=True))

web_index_page.add_item(shc.web.widgets.Switch("Ceiling Lights")
                        .connect(ceiling_lights))


# Store timestamp of last change of the ceiling lights in a Variable (via logic handler) and show
# it in the web user interface
ceiling_lights_last_change = shc.Variable(
    datetime.datetime, "ceiling lights last change",
    initial_value=datetime.datetime.fromtimestamp(0))

@ceiling_lights.trigger
@shc.handler()
async def update_lastchange():
    await ceiling_lights_last_change.write(datetime.datetime.now())

web_index_page.add_item(
    shc.web.widgets.TextDisplay(datetime.datetime, "{:%c}", "Last Change of Ceiling Lights")
    .connect(ceiling_lights_last_change))


# close shutters via button in the web user interface (stateless event, so no Variable required) 
web_index_page.add_item(
    shc.web.widgets.ButtonGroup(
        "Shutters",
        [
            shc.web.widgets.StatelessButton(shc.interfaces.knx.KNXUpDown.DOWN,
                                            shc.web.widgets.icon("arrow down"))
                .connect(knx_connection.group(shc.interfaces.knx.KNXGAD(3, 2, 1), dpt="1.008"))
        ]
    ))

# use expression syntax to switch on fan when temperature is over 25 degrees 
temperature = shc.Variable(float, "temperature")\
    .connect(knx_connection.group(shc.interfaces.knx.KNXGAD(0, 0, 1), dpt="9", init=True))
fan = shc.Variable(bool, "fan")\
    .connect(knx_connection.group(shc.interfaces.knx.KNXGAD(0, 0, 2), dpt="1"))\
    .connect(temperature.EX > 25.0)

# Start up SHC
shc.main()

License

Smart Home Connect is published under the terms of the Apache License 2.0.

It's bundled with multiple third party works:

• “Prism” – Subtle Patterns by Toptal Designers (Creative Commons BY-SA 3.0)

See LICENSE and NOTICE file for further information.

Dependencies

SHC depends on the following Python packages:

• aiohttp and its dependencies (Apache License 2.0, MIT License, Python Software Foundation License, LGPL 2.1, 3-Clause BSD License)
• jinja2 and MarkupSafe (BSD-3-Clause License)

Additional dependencies are required for some of SHC's interfaces. They can be installed automatically via pip, by specifying the relevant 'extras' flag, e.g. pip install smarthomeconnect[mysql] for mysql logging support.

• Logging via MySQL [mysql]:
  • aiomysql and PyMySQL (MIT License)
• KNX interface [knx]:
  • knxdclient (Apache License 2.0)
• DMX interface [dmx]:
  • pyserial-asyncio & pySerial (BSD-3-Clause License)
• MIDI interface [midi]:
  • mido (MIT License)
  • python-rtmidi (MIT License) incl. RTMidi (modified MIT License)
• MQTT interface [mqtt]:
  • paho-mqtt (Eclipse Public License v1.0 or Eclipse Distribution License v1.0)
  • asyncio-mqtt (BSD-3-Clause License)
• Pulseaudio interface [pulse]:
  • pulsectl and pulsectl-asyncio (MIT License)
• Telegram Bot interface [telegram]:
  • aiogram (MIT License) + its dependencies (BSD-3-Clause License + MPL 2.0)
• file-based persistence storage [file_persistence]:
  • aiofile & caio (Apache License 2.0)

In addition, the following Javascript libraries from NPM are required for the web UI frontend. They are not included in this repository or in source distribution packages of SHC. Instead they are downloaded and packed during Python package build and bundled in the "binary" Python packages (including "wheel" packages) of SHC:

• Fomantic UI CSS framework (MIT License)
• jQuery (MIT License)
• iro.js (Mozilla Public License 2.0)
• Chart.js (MIT License)

Development

Feel free to open an issue on GitHub if you miss a feature or find an unexpected behaviour or bug. Please, consult the documentation on the relevant topic and search the GitHub issues for existing reports of the issue first.

If you want to help with the development of Smart Home Connect, your Pull Requests are always appreciated.

Development setup

Setting up a dev environment for SHC is simple: Clone the git repository and install the development dependencies, listed in requirements.txt (+ the python-rtmidi module if you want to run the MIDI tests). These include all dependencies of smarthomeconnect with all extras:

git clone https://github.com/mhthies/smarthomeconnect
cd smarthomeconnect
pip3 install -r requirements.txt
pip3 install python-rtmidi

You may want to use a virtual environment to avoid messing up your Python packages.

Web UI Frontend Assets

Additionally, you'll need NodeJS and NPM on your machine for downloading and packing the web UI frontend asset files. Use the following commands to download all frontend dependencies from NPM and package them into /shc/web/static (using Parcel.js):

npm install
npm run build

When working on the web UI source files themselves (which are located in web_ui_src), you'll probably want to run Parcel.js in monitor mode, providing automatic re-packing and reload on every change:

npx parcel web_ui_src/main.js --dist-dir shc/web/static/pack --public-url ./

Tests and Code Style

Please make sure that all the unittests are passing, when submitting a Pull Request:

python3 -m unittest

The web tests require Firefox and geckodriver to be installed on your system and the frontend assets.

Additionally, I'd like to keep the test coverage on a high level. To check it, you may want to determine it locally, using the coverage tool:

coverage run -m unittest
coverage html
# open htmlcov/index.html

We also enforce static type correctness with MyPy and Python codestyle rules with flake8. To run the static type checks and codestyle checks locally, simply install MyPy and flake8 and execute the mypy and flake8 commands in the shc project repository.

All these checks are also performed by the GitHub Actions CI for Pull Requests and the master branch.

smarthomeconnect's Issues

Add client interface library for websocket API

web: Improve layout of HideRow

Add InfluxDB interface for data logging

examples: Create example of complex logics

Including expressions, logic handlers, internal Variables, BreakableSubscription, Hysteresis, and TimerSwitch.

Create file-based persistence store

The persistence store shall allow to store the current values of connected objects in a file or directory on the harddrive to persist them across restarts of the SHC application and the host computer. Therefore, it shall provide any number of named, readable + writable objects which write every received value update to the persistence file.

Optimally, changed values are stored immediately to persist them even with unexpected crashes/shutdowns. Still, the implementation should be somehow optimized to avoid parsing, updating serializing and rewriting the whole persistence store on each value update.

Fix LogList web widget

• Correct reconnect behaviour
• Fix styling
• Allow value and timestamp formatting

Build JS/CSS dependencies from source and minify custom CSS/JS code

This should be done automatically when packaging the Python code (ref. #5).

Add web ui gauge widget

Maybe something like this:

Create "stripe chart" log widget for boolean and enum variables

Idea: A bar of colored stripes, one stripe for each value change, with its color representing the value (True/False or an enum value) and its with representing the duration until the next value chage.

Current implementation idea: Use Chart.js' API to create a new chart type, possibly based on a stacked bar chart.

Follow up to #9

log: Refactor log widgets in to web module

Add error propagation to write() implementations

This would enable showing error messages in the user interface and report errors back to interfaces (such as the HTTP API) when value update propagation to one or more interfaces failed.

Requires reasonable _write() timeouts in interfaces.

examples: Create example for using position of sun and weather forecast

web: Use importlib to load static web ressources

Goal is, to make SHC "zip_safe", i.e. make it executable directly from an unextracted wheel-package. This requires to load ressource files in a way that transparently extracts them from from the package if required, e.g. importlib.ressources.

For reference, see

• https://setuptools.pypa.io/en/latest/userguide/miscellaneous.html#setting-the-zip-safe-flag
• https://setuptools.pypa.io/en/latest/userguide/datafiles.html#accessing-data-files-at-runtime

However, I'm not sure if this is possible with our current approach of serving our static web ressources from a single static directory (see shc/web/interface.py:112).

Add monitoring script for evaluating the /monitoring endpoint in Nagios (or compatible monitoring systems)

The monitoring script should be a single-file Python 3 script, only requiring on the Python standard library, if possible. It shall comply with the Nagios Plugin API (text output and exit code): https://assets.nagios.com/downloads/nagioscore/docs/nagioscore/3/en/pluginapi.html
It shall provide command line parameters to configure the URL of the SHC server to be monitored and aspects of the server to be monitoried. Supported aspects should be:

• the overall SHC server, representing the overall status as calculated by the SHC server, based on the interface criticality
• only the SHC server itself (i.e. is the server reachable and returns valid JSON data) – similar to monitoring it with the check_http script
• only a single SHC interface (out of the provided JSON data)
  • should return the interface's metrics as Nagios perfdata

interfaces: Reconnect backoff timeout should be resetted upon successful connection

SupervisedClientInterface implements an exponential backoff timeout before reconnect, which is used by interfaces like shc_client, mqtt and others. However, the exponential backoff time is not resetted, once the connection can successfully be re-established, resulting in a overly long reconnection timeout when the next interruption occurs.

shc_client: Only subscribe to API objects if local subscribtions exist

Allow timers to be dynamically disabled/enabled

Maybe add an attribute enabled: bool and method enable(enabled: bool) -> None to all timers.
An additional wrapper class, inheriting from Readable[bool] and Writable[bool], could transform this into a connectable object.

It could also be made Reading[bool] to allow initialization from persistence.

Create log plot widget for numeric log variables

Requires #7

Current approach: Use Chart.js

Allow writhing logic handler functions without (or just one) parameters

In most cases, logic handler functions do not need the origin argument or even the value argument. (This is because the logic handlers are typically triggered by a timer or by multiple Subscribable objects, so the current values of relevant variables need to be read within the logic handler anyways.) Thus, it would be nice if the value and origin parameters could simply be omitted when defining a logic handler function.

I.e. If a logic handler is defined without parameters, it will automatically be triggered/called without arguments. If a logic handler is defined with just one parameter, the triggering value will be passed as a single argument, but the origin list will be omitted.

Implementation idea: This magic dropping of arguments should be implemented in the @shc.handler() decorator. We could examine the decorated function using inspect.signature() (or so) and later call the logic handler in an appropriate way, when triggered.

Add @expression decorator to transform simple function into ExpressionHandler

web: Add image ui widget with placeable widgets

web: Remove server-side template rendering

The web widgets use client-side JavaScript for dynamically changing the UI from received object state. Some of them already generate most of the HTML layout of the widget in JavaScript as well, e.g. by using a third-party JS library (like Iro or Chart.js). The other widgets use server-side template rendering with Jinja2 to generate the required HTML elements. However, these HTML templates are all quite simple, so we could reduce complexity by removing the template rendering and let all widgets' JavaScript code generate the HTML elements.

We can keep the base.htm template for rendering the overall web page with placeholders for each widget and an embedded JSON structure containing the specification of the individual widgets or move to a full single-page application, where the specification of the page layout and widget configurations is fetched as JSON from an API endpoint and all HTML rendering is done in client-side JavaScript.

tasmota: Combine energy monitoring connectors to a single tuple-typed one

This will allow to process all individual measurments (current, voltage, power, powerfactor, …) as a single value update, e.g. to write it into a single InfluxDB row. For individual processing, the tuple can be split using an UpdateExchange with its fields.

web: Add optional confirmation dialog for ui action widgets

Change Interface.get_status() to return a subscribable status connector

It should be possible to subscribe to each interface's status if the interface supports it. The status connector should be at least Readable[InterfaceStatus] and optionally Subscribable[InterfaceStatus]. The normal HTTP-based monitoring will read from these status connectors, whereas subscribing allows to react to interface status changes.

web: Add support for HTML title suffix

Should be an optional parameter for shc.web.interface.WebServer, taking a suffix string, format string or formatting function for the title.

examples: Create web ui standalone example without external interfaces

Add Redis interface

The Redis interface should allow creating Readable + Writable connectors for using Redis String-values associated to a specific key for persistence.

In addition, the interface could allow reading and writing to/from Redis streams, possibly allowing to use Redis as a datalog store or communication interface with other applications.

Add Icinga2 API interface

The interface should allow to create connectors for reading the state of hosts or services, monitored by Icinga.

(I'd like to have LED lights in my appartment, changing from green to red when one of my web servers goes bad.)

knx: clear origin in local feedback when sending value to group address

In KNX GroupVariable objects we re-publish each value update which is written to the object, effectively creating a "local feedback". This way, other objects subscribed to the GroupVariable receive the value update as it came from the KNX bus. However, we keep the origin list intact, such that the value update is not written back to the orginating object (or any other object in the update's path).

This behaviour helps avoiding feedback loops, but it poses two problems:

1. It will be inconsistent with the MQTT interface, where we cannot prevent feedback (without origin reference), since the MQTT broker will return our sent messages without any hint about their origin.
2. When a concurrent, conflicting update comes from the KNX bus, it will result in a inconsistent state of KNX bus devices and our internal variable (and even devices connected via other interfaces). (see also #27)

Thus, we should probably clear the origin value for the local value feedback, i.e. make it look like the value came straigt from the KNX bus.

Add MQTT interface (incl. JSON-over-MQTT)

Ideally, we aim to be compatible with HomeAssistant's MQTT protocol integrations and ESPHome

iPhone / Android interface

I've seen the Web-UI and was wondering whether there is any kind of iPhone or Android App to control the widget via smarthomeconnect?

BTW I love your approach of handling the house automation in pure python scripts. I have no need for local UIs etc. to maintain the system. I've been using openhab now for 10 years and I'm looking for a more adequate developer solution. smarthomeconnect looks very promising, great work 👍 But a smart phone interface is a must, especially for WAF 😄

Add automatic reconnect to interfaces

Add CalDAV interface

The CalDAV interface should be able to read a specific calendar from a CalDAV calendar server. It should allow to create Readable + Subscribable bool-typed connector objects that tell whether currently any event, matching a specific pattern, is running according to the calendar. The connector-specific patterns could allow to consider only busy (not free) events, or only events having their summary or description matching a specific string pattern.

In addition, the interface could allow to create similar connectors that return the datetime timestamp of the next matching event.

Add simulation mode to interfaces to allow offline tests

Ramp generators: Interrupting a ramp with a new target value can result in an incomplete ramp

I experienced this with the RGBWHSVRamp: When fading from one white-ish color to another color and interrupting this ramp with black, the ramp seems to miss the last step sometimes, such that black is never reached completely.

web: Switch widget changes visual state incorrectly when confirmation is denied

Make Python package dependencies of interfaces optional

• Don't import interface modules automatically
• Use feature_requires in setup.py to make dependencies optional

examples: Create example for KNX specifics

Including usage of TwoWayPipe and stateless connections

Make stateful objects Reading to allow initialization from persitance

• Hysteresis
• AbstractRamp (? – for really long ramps?)

expressions: Support string and bytes operations

web: Add support for serving and including favicon

Should be implemented as an option for shc.web.interface.WebServer, taking a file path of the icon file.

web: don't start websocket on /monitoring page

Currently, the Javascript for active elements on the web page is loaded by default for every page based on our base.html template, including the monitoring page. However, the monitoring page does not provide dynamic contents (all values are server-side rendered into the template), so the Javascript is not necessary.

As a solution, we could call the on-load JS code on normal SHC UI pages explicitly as a JS function from the HTML template. A Jinja template variable could be used to enable/disable calling this function.

Additionally, we could provide an auto-reload feature for the monitoring page.

Add setup.py for packaging and installation

Add a SceneManager utility class

The idea is to create SceneManager objects, that somehow behave like KNX scenes: You connect a number of connectable objects to the SceneManager (which provides any number of subscribable + reading connectors). Than you can call a method to store the current value of the connected objects as a "scene" with a specific number. Later, you can call another method to recall a stored scene, i.e. publish the stored value to all connected scenes.

The SceneManager should ensure persistence of the stored scenes somehow, e.g. using the JSON-representation of the stored values and writing it to a file (or some other persistent store).

The interface would look like this:

class SceneManager:
    def __init__(persistence_file: Path): ...
    def connector(type_: Type[T], name: str) -> SceneConnector: ...
    async def store_scene(scene_number: int) -> None: ...
    async def recall_scene(scene_number: int) -> None: ...

class SceneConnector(Reading[T], Subscribable[T], Generic[T]): ...

Maybe, there might also be a current_scene connector which allows to read, write and subscribe to the currently selected scene as an int value. In this case, it would also be nice if the SceneConnectors would be writable as well and the scene manager would update the current_scene value if the current values of the connected objects (coincidentally) resemble a specifc stored scene.

specify and implement log variable aggregation

examples: Create complex example with multiple modules

shc_client: Conflicting updates from server and client can result in inconsistent states

We should introduce a republishing of server-to-client-updates at the client side, when we are waiting for the server to acknowledge a concurrent client-to-server update. Similar to the origin-resetting mechanism for (stateful) Subscribable objects within SHC, this should resolve the state inconsistency.

Simultaneous updates of connected variables may cause inconsistent states

When multiple variables are connectected (e.g. in server-client scenarios) and two value updates arrive at the same time (from "different sides"), the value updates may probably "cross over" and result in inconsistent states and different orders of outgoing updates.