kylebenson/scale_client

The Python-based SCALE Client software for acquiring data from various sensors (i.e. via Raspberry Pi platform), processing it, and sharing it through multiple networks and data exchange protocols.

scale_client's Issues

Storing sensor data locally

The event reporter should store data locally (SD card?) in addition to reporting it via the Internet. This should essentially be a queue, where old data is archived and then eventually removed to make room when a certain database size is reached (configurable parameter). This should especially work when in an offline mode, and so should keep track of whether data has been reported or not for when regaining network connectivity.

VirtualSensor should support threshold/condition for use by all children

A lot of the PhysicalSensor implementations make use of a 'threshold' parameter that's used for conditionally reporting events when they're sampled from the sensor. Instead of each class implementing this logic on its own, we could easily have this be a configurable parameter in VirtualSensor so that all the others can inherit it. Furthermore, it could easily support different conditional operators other than just a 'greater than threshold' one that most use.

Consolidating circuits logic and providing alternatives

The circuits-based implementation has some logic spread across multiple classes. They should all be consolidated as much as possible. Also, the classes should be refactored into abstract base classes and default implementation ones (e.g. from application import Application) should be for circuits. Additionally, adding a purely threading-based version (like original client) could be useful if we run into more bugs with circuits. Would also allow for potentially using something like asyncio or Twisted in the future...

Specifically, we need to refactor the following classes:

• SensedEvent relies on circuits.Event (relayed_sensed_event too)
• VirtualSensor and some other sensors have a bunch
• Application has (should have) most (all?) of it
• Broker is just a circuits.Manager

Watchdog daemon

Add a watchdog daemon to make sure that the SCALE service is always up and running. Perhaps using something like http://linux.die.net/man/8/watchdog ???

MQTTEventSink to use paho-mqtt in place of mosquitto

The Python mosquitto project was donated to paho 2 years ago. The python-mosquitto package seems to be out-dated. I had trouble using it for ca file based encryption, while I succeeded in paho. I think at some point we will need to change the corresponding code in MQTTEventSink to use paho in place of mosquitto.

Consolidate all mesh-related code

Currently the mesh-networking-related code is spread across a lot of different files with small poorly-documented hacks in various places. Ideally, all of that code should be in a scale_client.network.mesh package.

Sensor.type() should be Sensor.get_type()

type is a Python keyword and so this function should be refactored to be get_type()

Alternative event encoding

Rather than have all SensedEvents formatted in the same heavy JSON format with all those fields, add an ability to configure the client to send in a) a format other than JSON and b) using an alternative schema (perhaps based on a template?) so as to only include some of the fields.

Virtual sensor plumbing

We need a method for raw sensors to send data to virtual sensors, which will perform some analytics / event detection before creating sensed events, as well as directly put raw sensed events into the event queue. This should ideally utilize some existing light-weight streams library to simplify our code and avoid reinventing software to maximize the system's efficiency.

EventReporter logic extensions

Since we want to be able to add custom code to the EventReporter class for e.g. forwarding certain SensedEvents out over the mesh relay, we will need to support a customizable EventReporter. This could take the form of a pluggable helper class that determines how to handle individual events or it could be a subclass that overrides some of the parent class's logic to perform these custom handling functions (even both?).

Network manager module

More discussion is required on exactly what this will be, but it should monitor network connections and support switching over to ad-hoc mode/routing packets to Sigfox-enabled devices.

Raspberry Pis are flooding "motion_detected" events

I don't think the client will flood this event when the PIR sensors is not attached. (Tested here.)

Therefore there might be a weird issue on those Pis.

Raw-value analog virtual sensors should return percentage instead of integer values

Analog sensors that return raw values should read percentage like 0.0 ~ 100.0, instead of integer values like 0 ~ 1023 (current implementation).
Therefore if we get different or more accurate ADCs we don't need to rewrite second-level virtual sensors like TemperatureHighVirtualSensor, LightFlashVirtualSensor, etc.

Not urgent, but should change at some point, and upgrade all devices together in order to avoid confusion on sensor readings.

Client remote control API

Once CoAP is included, we could create an external-facing API for us to e.g. adjust sensor sampling rates, issue commands, etc.

Configuration file

We need a configuration file (probably in yaml format) that tells the system what sensor devices it should expect to be attached to the system, what thresholds they will report at, how often they sample, what brokers they use and where those connect, etc.

This will replace the multiple client files such that one client is run, parses this file, and determines which sensors to connect to from the configuration contents.

Need a modification in README.md

In README.md, in the Quickstart section is a list of commands. Between lines 2 and 3, it seems that "cd scale_client" is needed.

SigFox publisher needs to be updated after event type redefinition

I think the SigFox publisher now uses the old event type to distinguish among different events. Now it needs to be updated or it reports a lot of "Unknown events".

Event types now becomes fewer - one for every virtual sensor. I am not sure if SigFox events should also follow this change.

Will it be good that SigFox publisher only publish high priority events?

Collapse all DummyXX classes into one DummyVirtualSensor

Pretty straightforward. Its constructor could take the type, priority, value, and something specifying how often to send an event. Later the value and how often could be more than just constants (stats distributions?).

Real-time QoS guarantees for event reporting

We need to implement some form of real-time QoS guarantees in the EventReporter. That is, if enough events are generated within the system, we need to pick only the highest priority ones to send such that we don't create an infinite backlog of unreported events. This is especially important for the Sigfox publisher, whose ultra-low bandwidth capabilities mean that we likely won't be able to report every single event generated but should only report higher-priority events (we should also guarantee that we don't create DUPLICATE EVENTS when they are also reported via MQTT).

We can still store the unreported messages in local storage and send them again later when there are fewer events being generated and/or we have more bandwidth available, but let's save that for a future enhancement and just get the initial real-time system working for now.

Test framework

We have no testing framework! We need one to ensure robustness of the code and prevent a failed demo. It should exercise the full end-to-end (sensor-to-cloud) pipeline, including various sensor readings and network disconnection.

Geo-location tagging feature

Some mechanism for tagging SensedEvents with geo-location data. Two options:

1. Gather info from online service based on IP address and perturb them slightly (possibly zeroing in on known locations first, e.g. DBH, Thingstitute, etc.) so that not all markers are on the same exact point.
2. Assign specific rooms to each SCALE box. This could either be stored as a value in the local config file or it could be in a database somewhere, which will require rethinking how the data flows to the dashboard as it will either have to have this tag when it arrives or the dashboard will have to do this lookup itself (which is probably the easiest? just a RESTful endpoint on the SCALE server...)

SensedEvents should obscure the dict/json representation of event data underneath

I already added a get_raw_data() function, but I think we need more abstractions for this fairly complex object. It's become fairly clunky and has its logic spread out over a lot of different places that are now requiring checks to validate it's correct state. In particular, I'm looking at how the "condition" part has to be duplicated in the VirtualSensors (once for the logic itself, another time to record how the logic was done). Wouldn't it be great to abstract this into one text representation, configurable via the config files, that is parsed into an actual logical rule and then executed in the VS?

I modified VSs for the whole read_raw_data() vs read() thing to start addressing some of this wonkiness where sometimes we want to work in raw sensor readings and other times in the whole object.

I've also run into issues of the whole TOPIC concept and whether it belongs inside of a SensedEvent or outside as a separate concept (maybe ONLY pertinent to MQTT?). This will also ease omitting unchanged details between sensor readings, compression, etc.

It may be time to bring in some more advanced Python modules that add nice syntax and translations for these operations, such as for database support.

Setting daemon=True for classes when constructing?

I'm not sure what this is doing, but it should really be handled from inside the class constructor itself rather than from the outside.

Organize SCALE client directory

File for SCALE client should be broken up into logical submodules such as sensors, publishers (change this name while you're at it), and (possibly) events (once we have more than just the one sensed event).

RPi event types should be more general

The current event types in the RPi sensors look like "SCALE_raw_Gas_RPi", but should really only look like "explosive_gas". This extra information only complicates our data schema and in fact makes analysis incredibly difficult as it does not allow for generalizing across devices.

All of the sensors should be updated to reflect this change and use the get_type() function instead of hard-coded values (see #13).

Debug statements

We need debug statements throughout the code ASAP! Ideally, we should be able to turn on varying levels of verbosity, but we at least need to be able to see that MQTT connects ok and when messages are being sent so we know when and what to expect to receive in a client.

`setuptools` fails to copy data files

It is noticed that (the current version of) setuptools fails to copy data files. However the writing test, which is conducted before appending the data file to the list of data files to copy, is always successful (no error message is printed).

Should figure out what is wrong here.

InternetManager to determine if we have internet access

Since many different components in the SCALE client system care about the status of the network connection, we should monitor this in some way (pinging some server? listening for disconnected sockets in MQTT, etc.?) and locally publish a network_(dis)connected event when the status changes so that other components can react accordingly. For example, the MQTTEventSink should always return "False" when asked check_available() so that the EventReporter doesn't block the entire client waiting for the Sink to time out while trying to reconnect.

Install as a package / service

All of the code should be packaged in such a way that it can be easily installed (setup.py), perhaps via pip, and set to run on startup as a service. Note that this relies on first completing #1.

Error during Application __init__ doesn't prevent it from running

If we encounter some exception while creating an Application/Sensor (e.g. a ValueError is raised in one of the init functions to indicate incorrect parameters), this doesn't stop the Application from running. This is likely due to circuits registering the Application and then starting it when the whole system (top-most Component) is started. Perhaps we need to specify a del method to unregister it when it isn't created properly? Or another suggestion on the web is to make use of new to catch exceptions and handle the error.

Some virtual sensor policies modifed for test purpose

Nalini asked us to collect raw data for today's test which lasts for several hours. Therefore, the policies of some virtual sensors has been modified to report every raw data readings.
The affected sensors are:
temperature_virtual_sensor.py
gas_virtual_sensor.py
light_virtual_sensor.py
Need an elegant way of doing this.

Failures with absolute imports

PyCharm resulted in absolute imports after Kyle's re-factory. These absolute imports cannot be resolved by Python 2. As a consequence, the clients no longer work if we attempt to run them like this:

cd /home/pi/SmartAmericaSensors/scale_client/core
./client_computer.py

Setting up environment variables may help solve this problem. We can run the clients with

cd /home/pi/SmartAmericaSensors/scale_client/core
env PYTHONPATH=/home/pi/SmartAmericaSensors ./client_computer.py

However, since our client will be registered as a daemon (also known as a Linux service) and will be called using start-stop-daemon, where environment variables are discarded, the solution above won't work in real deployments.

Tested working:

1. Having the following code paragraph at the beginning of the client file:

import sys
sys.path.append("/home/pi/SmartAmericaSensors"); # Path hard-coded in script

Tested NOT working:

1. Having PYTHONPATH set-up in /etc/environment
2. Having PYTHONPATH set-up in the daemon configuration file (/etc/init.d/scale).
3. Having env PYTHONPATH=/home/pi/SmartAmericaSensors before start-stop-daemon in the daemon configuration file.
4. Running the client with Python 3 (python3)

Not tested:

1. Using from __future__ import absolute_import in scripts

CEL_DAEMON_PATH is wrong

this should be temperature-streams not temprature-streams. Also, what is CEL supposed to stand for? Furthermore, the _daemon_path is never even used in temperature_virtual_sensor.py

Sigfox Sleep Cause EventReporter Blocked

Phenomenon:
We observed that, the messages that were sent by SheevaPlug had a fixed interval. Also, there was serious delay between real events happened and corresponding messages were sent.

Reason:
SigFox sleep function cause EventReporter blocked per second(depends on sleep length). Sigfox sleep function was used after send() and before receive() that were called by EventReporter to wait for Sigfox adapter return message. However, this will block the whole EventReporter, and MQTT sending is also influenced.

Temporary Solution:
Simply delete sleep function before calling receive on the Sigfox publisher, but this may cause the missing of Sigfox return value.

Possible Solution:
We can implement all the publishers as independent threads. Charles gave the idea that, instead of blocking and waiting for response, the response could be obtained by EventReport by Callback functions.

Identify Raspi by physical box

We should have some method of physically identifying which raspi device we're looking at when we open a box. We could either tape the last 6 of its MAC inside the black box somewhere or perhaps have a way of sending it a signal that causes one of the LEDs to flash.