Giter Site home page Giter Site logo

balena-io-experimental / edge-node-manager Goto Github PK

View Code? Open in Web Editor NEW
22.0 16.0 6.0 7.58 MB

Resin uC edge-node-manager written in Go

License: Apache License 2.0

Go 95.75% Shell 1.59% JavaScript 2.66%
resin dependent-devices microcontroller micro dfu ota bluetooth-low-energy gateway

edge-node-manager's Introduction

edge-node-manager

Go Report Card Build Status

resin.io dependent device edge-node-manager written in Go.

Getting started

  • Sign up on resin.io
  • Work through the getting started guide
  • Create a new application
  • Set these variables in the Fleet Configuration application side tab
    • RESIN_SUPERVISOR_DELTA=1
    • RESIN_UI_ENABLE_DEPENDENT_APPLICATIONS=1
  • Clone this repository to your local workspace
  • Add the dependent application resin remote to your local workspace
  • Provision a gateway device
  • Push code to resin as normal :)
  • Follow the readme of the supported dependent device you would like to use

Configuration variables

More info about environment variables can be found in the documentation. If you don't set environment variables the default will be used.

Environment Variable Default Description
ENM_LOG_LEVEL info the edge-node-manager log level
DEPENDENT_LOG_LEVEL info the dependent device log level
ENM_SUPERVISOR_CHECK_DELAY 1 the time delay in seconds between each supervisor check at startup
ENM_HOTSPOT_DELETE_DELAY 10 the time delay in seconds between hotspot deletion and creation
ENM_CONFIG_LOOP_DELAY 10 the time delay in seconds between each application process loop
ENM_CONFIG_PAUSE_DELAY 10 the time delay in seconds between each pause check
ENM_HOTSPOT_SSID resin-hotspot the SSID used for the hotspot
ENM_HOTSPOT_PASSWORD resin-hotspot the password used for the hotspot
ENM_BLUETOOTH_SHORT_TIMEOUT 1 the timeout in seconds for instantaneous bluetooth operations
ENM_BLUETOOTH_LONG_TIMEOUT 10 the timeout in seconds for long running bluetooth operations
ENM_AVAHI_TIMEOUT 10 the timeout in seconds for Avahi scan operations
ENM_UPDATE_RETRIES 1 the number of times the firmware update process should be retried
ENM_ASSETS_DIRECTORY /data/assets the root directory used to store the dependent device firmware
ENM_DB_DIRECTORY /data/database the root directory used to store the database
ENM_DB_FILE enm.db the database file name
ENM_API_VERSION v1 the proxyvisor API version
RESIN_SUPERVISOR_ADDRESS http://127.0.0.1:4000 the address used to communicate with the proxyvisor
RESIN_SUPERVISOR_API_KEY na the api key used to communicate with the proxyvisor
ENM_LOCK_FILE_LOCATION /tmp/resin/resin-updates.lock the lock file location

API

The edge-node-manager provides an API that allows the user to set the target status of the main process. This is useful to free up the on-board radios allowing user code to interact directly with the dependent devices e.g. to collect sensor data.

Warning - Do not try and interact with the on-board radios whilst the edge-node-manager is running (this leads to inconsistent, unexpected behaviour).

SET /v1/enm/status

Set the edge-node-manager process status.

Example

curl -i -H "Content-Type: application/json" -X PUT --data \
'{"targetStatus":"Paused"}' localhost:1337/v1/enm/status

curl -i -H "Content-Type: application/json" -X PUT --data \
'{"targetStatus":"Running"}' localhost:1337/v1/enm/status

Response

HTTP/1.1 200 OK

GET /v1/enm/status

Get the edge-node-manager process status.

Example

curl -i -X GET localhost:1337/v1/enm/status

Response

HTTP/1.1 200 OK
{
    "currentStatus":"Running",
    "targetStatus":"Paused",
}

GET /v1/devices

Get all dependent devices.

Example

curl -i -X GET localhost:1337/v1/devices

Response

HTTP/1.1 200 OK
[{
	"ApplicationUUID": 511898,
	"BoardType": "esp8266",
	"Name": "holy-sunset",
	"LocalUUID": "1265892",
	"ResinUUID": "64a1ae375b213d7e5af8409da3ad63108df4c8462089a05aa9af358c3f0df1",
	"Commit": "16b5cd4df8085d2872a6f6fc0c378629a185d78b",
	"TargetCommit": "16b5cd4df8085d2872a6f6fc0c378629a185d78b",
	"Status": "Idle",
	"Config": null,
	"TargetConfig": {
		"RESIN_HOST_TYPE": "esp8266",
		"RESIN_SUPERVISOR_DELTA": "1"
	},
	"Environment": null,
	"TargetEnvironment": {},
	"RestartFlag": false,
	"DeleteFlag": false
}]

GET /v1/devices/{uuid}

Get a dependent device.

Example

curl -i -X GET localhost:1337/v1/devices/1265892

Response

HTTP/1.1 200 OK
{
	"ApplicationUUID": 511898,
	"BoardType": "esp8266",
	"Name": "holy-sunset",
	"LocalUUID": "1265892",
	"ResinUUID": "64a1ae375b213d7e5af8409da3ad63108df4c8462089a05aa9af358c3f0df1",
	"Commit": "16b5cd4df8085d2872a6f6fc0c378629a185d78b",
	"TargetCommit": "16b5cd4df8085d2872a6f6fc0c378629a185d78b",
	"Status": "Idle",
	"Config": null,
	"TargetConfig": {
		"RESIN_HOST_TYPE": "esp8266",
		"RESIN_SUPERVISOR_DELTA": "1"
	},
	"Environment": null,
	"TargetEnvironment": {},
	"RestartFlag": false,
	"DeleteFlag": false
}

Supported dependent devices

Further reading

About

The edge-node-manager is an example of a gateway application designed to bridge the gap between Resin OS capable single board computers (e.g. the Raspberry Pi) and non Resin OS capable devices (e.g. micro-controllers). It has been designed to make it as easy as possible to add new supported dependent device types and to run alongside your user application.

The following functionality is implemented:

  • Dependent device detection
  • Dependent device provisioning
  • Dependent device restart
  • Dependent device over-the-air (OTA) updating
  • Dependent device logging and information updating
  • API

Definitions

Dependent application

A dependent application is a Resin application that targets devices not capable of interacting directly with the Resin API.

The dependent application is scoped under a Resin application, which gets the definition of gateway application.

A dependent application follows the same development cycle as a conventional Resin application:

  • It binds to your git workspace via the resin remote
  • It consists of a Docker application
  • It offers the same environment and configuration variables management

There are some key differences:

  • It does not support Dockerfile templating
  • The Dockerfile must target an x86 base image
  • The actual firmware must be stored in the /assets folder within the built docker image

Dependent device

A dependent device is a device not capable of interacting directly with the Resin API - the reasons can be several, the most common are:

  • No direct Internet capabilities
  • Not able to run the Resin OS (being a microcontroller, for example)

Gateway application

The gateway application is responsible for detecting, provisioning and managing dependent devices belonging to one of its dependent applications. This is possible leveraging a new set of endpoints exposed by the Resin Supervisor.

The edge-node-manager (this repository) is an example of a gateway application.

Gateway device

The gateway device runs the gateway application and has the needed on-board radios to communicate with the managed dependent devices, for example:

  • Bluetooth
  • WiFi
  • LoRa
  • ZigBee

Throughout development a Raspberry Pi 3 has been used as the gateway device.

edge-node-manager's People

Contributors

brownjohnf avatar hedss avatar jbaldwinroberts avatar lucianbuzzo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

edge-node-manager's Issues

Improve DB implementation

Add the following functionality:

  • Read and write single field rather than the whole device []byte
  • Local cache to avoid unnecessary reads and writes

The main aim is to allow continual saving of state throughout the main process so that data is not lost if the gateway loses power or an error occurs

Automatically set the dependant application board type

Currently the dependant application board type has to be set manually. I want to automate this in the short term using the dependant application fleet configuration to set the board type.

In the long term the dependent application board type should be set when creating the dependant application, rather like the dependant application name is now.

This will allow us to remove this section.

Can't dial

4f4a7f70b5e201]] URL="http://127.0.0.1:48484/v1/devices/d6f80e5c7b26a010534ab1dad42d14151cc6ef26be9a439f0bfe3b110a7dd6"
07.03.17 16:26:26 (+0000) INFO Starting update                               Attempt=1 Name=Left
07.03.17 16:26:26 (+0000) ERRO Update failed                                 Error="can't dial: Command Disallowed" Name=Left
07.03.17 16:26:26 (+0000) INFO Starting update                               Attempt=2 Name=Left
07.03.17 16:26:26 (+0000) ERRO Update failed                                 Error="can't dial: Command Disallowed" Name=Left
07.03.17 16:26:26 (+0000) INFO Starting update                               Attempt=3 Name=Left
07.03.17 16:26:26 (+0000) ERRO Update failed                                 Error="can't dial: Command Disallowed" Name=Left
07.03.17 16:26:31 (+0000) INFO Starting update                               Attempt=1 Name=Middle
07.03.17 16:26:31 (+0000) ERRO Update failed                                 Error="can't dial: Command Disallowed" Name=Middle
07.03.17 16:26:31 (+0000) INFO Starting update                               Attempt=2 Name=Middle
07.03.17 16:26:31 (+0000) ERRO Update failed                                 Error="can't dial: Command Disallowed" Name=Middle
07.03.17 16:26:31 (+0000) INFO Starting update                               Attempt=3 Name=Middle
07.03.17 16:26:31 (+0000) ERRO Update failed                                 Error="can't dial: Command Disallowed" Name=Middle
07.03.17 16:26:45 (+0000) INFO ----------------------------------------------------------------------------------------------------
07.03.17 16:26:45 (+0000) INFO Processing application                        Application=microbit Number of devices=3

Dependent device delete sometimes does not work

Dependent device delete occasionally results in:

27.10.16 12:56:48 [+0100] 2016/10/27 11:56:48 http: panic serving 127.0.0.1:44926: runtime error: invalid memory address or nil pointer dereference
27.10.16 12:56:48 [+0100] goroutine 148 [running]:
27.10.16 12:56:48 [+0100] net/http.(*conn).serve.func1(0x10832840)
27.10.16 12:56:48 [+0100] /usr/local/go/src/net/http/server.go:1491 +0x9c
27.10.16 12:56:48 [+0100] panic(0x324170, 0x1070c010)
27.10.16 12:56:48 [+0100] /usr/local/go/src/runtime/panic.go:458 +0x454
27.10.16 12:56:48 [+0100] github.com/resin-io/edge-node-manager/api.DependantDeviceDelete(0x483cf8, 0x10805ef0, 0x10766d00)
27.10.16 12:56:48 [+0100] /go/src/github.com/resin-io/edge-node-manager/api/handlers.go:71 +0x224
27.10.16 12:56:48 [+0100] net/http.HandlerFunc.ServeHTTP(0x39c240, 0x483cf8, 0x10805ef0, 0x10766d00)
27.10.16 12:56:48 [+0100] /usr/local/go/src/net/http/server.go:1726 +0x34
27.10.16 12:56:48 [+0100] github.com/resin-io/edge-node-manager/api.Logger.func1(0x483cf8, 0x10805ef0, 0x10766d00)
27.10.16 12:56:48 [+0100] /go/src/github.com/resin-io/edge-node-manager/api/logger.go:15 +0x88
27.10.16 12:56:48 [+0100] net/http.HandlerFunc.ServeHTTP(0x10712500, 0x483cf8, 0x10805ef0, 0x10766d00)
27.10.16 12:56:48 [+0100] /usr/local/go/src/net/http/server.go:1726 +0x34
27.10.16 12:56:48 [+0100] github.com/resin-io/edge-node-manager/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0x10716150, 0x483cf8, 0x10805ef0, 0x10766d00)
27.10.16 12:56:48 [+0100] /go/src/github.com/resin-io/edge-node-manager/vendor/github.com/gorilla/mux/mux.go:107 +0x240
27.10.16 12:56:48 [+0100] net/http.serverHandler.ServeHTTP(0x10718640, 0x483cf8, 0x10805ef0, 0x1083c200)
27.10.16 12:56:48 [+0100] /usr/local/go/src/net/http/server.go:2202 +0x150
27.10.16 12:56:48 [+0100] net/http.(*conn).serve(0x10832840, 0x4843f0, 0x10836880)
27.10.16 12:56:48 [+0100] /usr/local/go/src/net/http/server.go:1579 +0xde8
27.10.16 12:56:48 [+0100] created by net/http.(*Server).Serve
27.10.16 12:56:48 [+0100] /usr/local/go/src/net/http/server.go:2293 +0x470

"Dependent" misspelled in many places

Dependent is spelled as dependant in many places, including all over in the actual codebase. I think we should correct this sooner than later, and certainly before open-sourcing it.

OTA progress reporting

The edge-node-manager currently prints OTA progress to the dashboard if Debug logging is enabled.
Once the supervisor progress API is in place I need to send this progress to the API instead.

Timed out waiting for notification

ENM logs:
02.03.17 17:45:08 [+0000] ERRO Unable to process application
ID: 18534, Name: microbit, Board type: microbit, Commit: b0c34
b1f1cd, Target commit: b0c34f7ac4c82047125399bb9336264103b1f1c
OR_DELTA:1 RESIN_HOST_TYPE:microbit], File path: /data/assets/
b9336264103b1f1cd Errors=[Timed out waiting for notification]

Micro logs:
18.01.70 05:27:56 [+0000] INFO Starting update

Device was bricked after this

BLE radio cannot be freed after gatt has taken control

  • Completely exit the ENM and then start it again - perhaps with a manager application to ease this process.
  • Use a dbus based Bluetooth library that accepts multiple incoming clients (https://github.com/ecc1/ble)
  • Create the Bluetooth object/interface outside the ENM and pass it into the parts of the code that need access, this would force the user to also write their code in Go.

Pull down fields that can be changed from the dashboard

Some fields can be changed from the dashboard, these need to appear on the edge-node-manager in real time - currently a container restart is required.

  • Application config
  • Device name
  • Device target config
  • Device target environment

Tidy BLE library logging

The current paypal BLE library prints random bits of logging to the dashboard e.g.

2016/10/26 14:54:28 ignore l2cap signal:[ 0C 00 05 00 12 02 08 00 0C 00 18 00 00 00 90 01 ]

There is no easy way to turn this off other than forking the library and commenting out the print statements.

We could try moving to https://gitub.com/currantlabs/ble which is still active and fixes niggles with the paypal BLE library such as the logging issue above.

currantlabs/ble#17

#75

Set up CI test cases for packages that do not interact with the hardware

Currently there is no CI testing implemented. It would be good to implement CI test cases for all packages that do no interact with the hardware, currently these are:

  • API
  • Application
  • Board
  • Config
  • Database
  • Process
  • Supervisor

Steps to achieve:

  • Travis CI integration
  • Database population tool
  • Environment variables population tool
  • Supervisor API simulation tool (https://github.com/jarcoal/httpmock)
  • Test case units for each package

Improve error handling

The error handling leaves a lot to be desired. There are two main issues:

  • Some functions return []error whereas others return error which leads to boilerplate code []error{err} to convert error into []error
  • If an error is triggered mid was through the process any changes are not saved into the DB (Issue #54)

Add timeout to updates

I am not really sure how to implement this or if its really necessary when everything is working correctly. It would deal with the symptom but not the cause.

It would still be nice as it makes it impossible for the edge-node-manager to hang during a Bluetooth operation in the event the Bluetooth devices goes offline, out of range etc.

Dependent device status gets out of sync

Currently the edge-node-manager only sets the dependent device status when it changes between online and offline. This results in out of sync behaviour in the following cases:

  • Gateway goes offline - all the dependant devices stay in the prior state
  • Gateway restarts and the saved state has not changed - all the dependent devices stay in the prior state (PR #50)
  • Gateway goes offline and so do all the dep. devices, once the gateway comes back online the dep. devices do not. This is because the gateway only sends the dep. device is_online state if it has changed and as far as the gateway is concerned the devices have always been online

Improve general logging

Currently lots of duplication as Info logging prints the device name whereas debug logging prints the complete device.

Pluginification

The ENM runs alongside other user code, needs some kind of interface

Micro:bit support

Steps:

  • Create micro:bit repo with boilerplate code and a dockerfile for builder compilation
  • Create micro:bit sub type under nRF51822 (also subtype nRF51822 DK)
  • Update ENM readme

Micro:bit repo

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.