openhausio/backend

HTTP API for the OpenHaus SmartHome/IoT solution

backend's Issues

Add "Eventbus" for cross communication

The idea behind a "Eventbus" is, that components can communicate with each other, across multiple (sub-) processes.
This would allow to run plugins in a separate child process, which has still access to all wanted components/objects. Keeps data between plugins/processes in sync.

Remove color.js due to corruption

See more here: Marak/colors.js#285

More to read/view
https://github.com/Marak/faker.js/commit/2c4f82f0af819e2bdb2623f0e429754f38c2c2f2#diff-1550ec65ac92f65817fc28928dfef526912b5f52356ff43651369bae92f56031

https://www.bleepingcomputer.com/news/security/dev-corrupts-npm-libs-colors-and-faker-breaking-thousands-of-apps/

Update `docker-compose.yml` to include nginx as reverse proxy?

Add nginx to handle the serving for the frontend?
/auth & /api calls passed through nginx to the backend, / is serving the frontend (static files), everything else is invalid.

https://www.domysee.com/blogposts/reverse-proxy-nginx-docker-compose
https://www.bogotobogo.com/DevOps/Docker/Docker-Compose-Nginx-Reverse-Proxy-Multiple-Containers.php

Add cors origin "127.0.0.1" for plugin/external process access?

Modify CORS Origin to allow access/requests from localhost/loobkack interface, to allow local running services access to the API

Remove user password field from API calls

Currently, under some circustances the password is leaked over API calls. E.g in error messages when the schema validation failed. Iterate over all /api/users calls & fields and remove/set password to null?

{
    "error": {
        "code": "ERR_VALIDATION",
        "details": {
            "_original": {
                "_id": "611033f0acc919461c3a1962",
                "name": "name",
                "email": "[email protected]",
                "password": "$2b$12$0E0RXurg73rd2/OJRDWepOlGKtgnbiv38NAwOnIarQkYku0v8cnFS",
                "timestamps": {
                    "created": 1628451824939,
                    "updated": 1634108576122
                }
            },
            "details": [
                {
                    "message": "\"_id\" is not allowed",
                    "path": [
                        "_id"
                    ],
                    "type": "object.unknown",
                    "context": {
                        "child": "_id",
                        "label": "_id",
                        "value": "611033f0acc919461c3a1962",
                        "key": "_id"
                    }
                }
            ]
        },
        "timestamp": 1634108576122,
        "message": "Validation on dataset failed"
    }
}

`request.js` implement redirect following

Choose (right) License

Because of many reasons, we should choose the right license:
https://dbad-license.org/ ?

Just some links/stuff:
Marak/colors.js#302
https://choosealicense.com/appendix/
https://github.com/marak/Faker.js/
https://github.com/Marak/faker.js/commit/2c4f82f0af819e2bdb2623f0e429754f38c2c2f2#diff-1550ec65ac92f65817fc28928dfef526912b5f52356ff43651369bae92f56031

Add "pre-commit" as dev dependecie?

pre-commit is a pre-commit hook installer for git. It will ensure that your npm test (or other specified scripts) passes before you can commit your changes. This all conveniently configured in your package.json.

Ensure that linting and other pre-commit stuff is done: https://www.npmjs.com/package/pre-commit

Add "vault" component

To interact with various APIs that may need authentication, we need a way to store credentials and tokens.
This is the purpose of the "vault" component, that acts like a password safe for authentication stuff.

Database relation

• icon field (rooms / endpoints / devices?)
• in room schema
  • add array of endpoints

Create & Release v1.0.0 version

• Fix issues (tagged with mile stone v1.0.0 release)
• Release version 1
• Setup build toolchains
• Setup CI/CD pipeline

Add autodiscover for devices/services

Listen for ssdp broadcast messages in the local network.

• Create "ssdp" component to allow plugins to "subscribe" to such messages (NT/USN)
• Parse messages
• Autofetch LOCATION field/convert xml to json?
• Notifications on "ssdp:alive" or "ssdp:byebye"
• Active search for devices
• Listen for dhcp requests/answer for assignd IPs?
  -> Would allow to trigger scenes/command when a smartphone connets or many more

GitHub actions dont start/run

Investigate why the github actions dont start/run anymore.

Check if plugins file/folder exists before load/require

Check if a plugin file/folder exsits before to require/load.
Backends crash if file is not found:

$ npm run start

> [email protected] start
> ./node_modules/.bin/nodemon index.js

[nodemon] 2.0.12
[nodemon] to restart at any time, enter `rs`
[nodemon] watching path(s): *.*
[nodemon] watching extensions: js,mjs,json
[nodemon] starting `node index.js`
Starting OpenHaus v1.0.0...
Error in plugin: Error: Cannot find module '/home/marc/projects/OpenHaus/backend/plugins/d11e7f38-91cb-4f32-89e8-3452f624bb47/index.js'
Require stack:
- /home/marc/projects/OpenHaus/backend/components/plugins/class.plugin.js
- /home/marc/projects/OpenHaus/backend/components/plugins/index.js
- /home/marc/projects/OpenHaus/backend/index.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:962:15)
    at Function.Module._load (internal/modules/cjs/loader.js:838:27)
    at Module.require (internal/modules/cjs/loader.js:1022:19)
    at require (internal/modules/cjs/helpers.js:72:18)
    at Plugin.boot (/home/marc/projects/OpenHaus/backend/components/plugins/class.plugin.js:38:30)
    at /home/marc/projects/OpenHaus/backend/index.js:361:20
    at Array.forEach (<anonymous>)
    at /home/marc/projects/OpenHaus/backend/index.js:356:8
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/home/marc/projects/OpenHaus/backend/components/plugins/class.plugin.js',
    '/home/marc/projects/OpenHaus/backend/components/plugins/index.js',
    '/home/marc/projects/OpenHaus/backend/index.js'
  ]
}

Refactor system/component `.update` method

Refactor system/component to fix the following issues:

• #34
• #39
• #45

Update calls are not updating/manipulating the target object, instead the create a copy of the target and add the copy to the .items array.

Fix only .update method, or generally improve structure/code?
Processing TODO.md

Remove "filter" from helpers?

Use case?

Fix "@TODO" / "@NOTE"

Fix/update things labeled with @TODO / @NOTE

Add in postman tests for requests/response

For #5 it would be awesome to have some kind of http API testing.
Postman can do this. Create some tests there.

https://learning.postman.com/docs/sending-requests/examples/
https://learning.postman.com/docs/designing-and-developing-your-api/mocking-data/mocking-with-examples/
https://learning.postman.com/docs/sending-requests/requests/
https://assets.postman.com/postman-docs/sent-request-v8.jpg
https://community.postman.com/t/checking-a-schema-with-tv4/7575/2

Components `.items` array manipulation does not update/remove items

Items are new added to the array and not removed when deleted.

Express error handling does not work properly

https://github.com/OpenHausIO/backend/blob/dev/routes/index.js#L119:

TypeError: res.status is not a function
    at /opt/OpenHaus/backend/routes/index.js:119:17
    at Layer.handle [as handle_request] (/opt/OpenHaus/backend/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/opt/OpenHaus/backend/node_modules/express/lib/router/index.js:317:13)
    at /opt/OpenHaus/backend/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/opt/OpenHaus/backend/node_modules/express/lib/router/index.js:335:12)
    at next (/opt/OpenHaus/backend/node_modules/express/lib/router/index.js:275:10)
    at expressInit (/opt/OpenHaus/backend/node_modules/express/lib/middleware/init.js:40:5)
    at Layer.handle [as handle_request] (/opt/OpenHaus/backend/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/opt/OpenHaus/backend/node_modules/express/lib/router/index.js:317:13)
    at /opt/OpenHaus/backend/node_modules/express/lib/router/index.js:284:7

But should, example from express documentation: https://expressjs.com/de/guide/error-handling.html

Containerize Backend

Backend should be containerized for easier install

`.remove` does not return object/item, but mongodb result object

A call rooms.remove() method returns the mongodb result object:

{
    "result": {
        "n": 1,
        "ok": 1
    },
    "connection": {
        "_events": {},
        "_eventsCount": 4,
        "id": 14,
        "address": "127.0.0.1:27017",
        "bson": {},
        "socketTimeout": 5000,
        "host": "127.0.0.1",
        "port": 27017,
        "monitorCommands": false,
        "closed": false,
        "destroyed": false,
        "helloOk": true,
        "lastIsMasterMS": 1
    },
    "deletedCount": 1,
    "n": 1,
    "ok": 1
}

Change this, that the object/item is return

Improve schema handling

How to handle schema changes?

• What to do with the existing data in the database?
• Post update installer?
• How to handle renames/deletions

Possible solution is, to call validate() with the data from the database, to apply default values for new added fields.

Schema before update

const schema1 = Joi.object({
    name: Joi.string().required()
});

Schema after update:

const schema1 = Joi.object({
    name: Joi.string().required(),
    enabled: Joi.boolean().default(true)
});

Now pass each dataset through the validation method, and the new enabled field has the default value set.
Need to find a solution for deletions / renaming of fields.

General:

• #2

Improve connector

Improve connector.js
Supply as separate module? -> Find a way to keep "systems request.js": #14

Enable/Activate user component

• Fire up user component
• Check if user has a valid JWT token
• Protect API requests with JWT
• Allow user to login/logout
• etc...

After `.update(...)` call, item gets added (again) as last `.items` item

A update call of the API adds the updated item again in to the .items array.

run plugins in seperate child process

Plugins run in the main process.
Implement to run plugins as a separate child process.
Keep the hooks/events in sync across the all plugin processes.

Update API calls to `/api/rooms` does not set the name field correctly

A quick test expose that when you update only the room name, the name field is not set the the given value, but to "name",

{
    "_id": "6032794756b6681d420d1946",
    "name": "name",
    "timestamps": {
        "created": 1613920583473,
        "updated": 1634109143775
    }
}

Linting does not work on componets/plugins

.eslintrc.json ignorePattern "plugins" prevent linting from componets/plugins

Add "dummy"/demo datbase setup

For quick start, add a "dummy" dataset, that can be imported.
Like a mongodb dump.

Update response headers

Since most of the request to the backand will come from a browser, we will run into cors errors if we dont update the response headers;

This should atleast be added: to the response header:
Access-Control-Allow-Origin: '*'

Maybe also this:
Access-Control-Allow-Headers: '*'
Access-Control-Allow-Methods: 'GET, POST, PUT, DELETE, OPTIONS'

Drop dependencie "duplexer" and use `Duplex.from`?

I have just found out, that you can create a duplex stream with the Duplex.from method.
Should we drop the duplexer3 package and use native methods?

Found here after a question on stackoverflow: https://stackoverflow.com/q/70953564/5781499

The less dependencies, the better.
See color.js corruption story... #63

Add plugin "intents" field?

Plugins must declare what intents they have.
E.g. Hook into device/endpoint update chain, so they must/want to use the "device"/"endpoint" component.
A plugin then must declare that they want "access" to this components.

Switch to rethinkdb?

In order to be able to put plugins into separate child processes, use rethinkdb with its out-of-the-box "change streams"

Component init error handling not working proplery

Error is thrown, error event not handled correctly.
In the <root/index.js , the init_components methods,

component.events.on("error", (err) => {
                logger.error(err, `Component "${name}" error!`);
});

Is not fired, instead a exception is thrown, (which is caught in the try/catch block) and the process exit.

Improve linting

• Add rule to disallow undefined globals, like setTimeout, Buffer without requireing/importing the module (timers, buffers)
• Add node specific rules, pseudo snippet: extends:node in .eslintrc.json

collection.findOneAndUpdate option [returnOriginal] is deprecated and will be removed in a later version.

https://stackoverflow.com/q/52572852/5781499

(node:21313) [MONGODB DRIVER] DeprecationWarning: collection.findOneAndUpdate option [returnOriginal] is deprecated and will be removed in a later version.
(Use `node --trace-deprecation ...` to show where the warning was created)

Provide OCI, LXC/LXD images?

Provide images for OCI, LXC/LXD?

Create unit tests for helper functions

 if (!handler(prop, value, target, receiver)) {

Handler is undefined in observe.js

backend/helper/observe.js
  49:10  error  'prop' is not defined     no-undef
  50:20  error  'prop' is not defined     no-undef
  51:20  error  'prop' is not defined     no-undef
  51:43  error  'prop' is not defined     no-undef
  51:59  error  'handler' is not defined  no-undef
  59:22  error  'handler' is not defined  no-undef
  70:21  error  'handler' is not defined  no-undef

Autoupdate "postman.json"

Update postman.json: https://learning.postman.com/docs/writing-scripts/test-scripts/

Add "scene" components

A scene is a collection of commands.
E.g Its Evening, and you want to have all lights dimmed, some off/on.
This can be done with a scene, a collection of devices/endpoints "states"
Basically a "multi"/"batch" command execution.

Improve/create unit tests

Create new/update unit tests.

Joi nested schema validation does not work

Joi.array(thing.schema()) does not validate child/array items

Create build toolchain

• linting
• "compiling"
• minimizing
• remove comments from source
• prune dependencies
• tree shaking (https://rollupjs.org/guide/en/)?
• etc...

Improve "logger"

• Create "trace" method to inspect function trace/execution
• Work more with classes like in components
• Fix LOG_TARGET
  -> Apply only in dev mode

Vault timestamp is not updated when `encrypt` is called

Should the timestamp change when the .encrypt() method is called?
Setting new values can be considered as "update"

Update (1

{
    "_id": "6199783105707e5faa429368",
    "name": "ZigBee Gateway",
    "identifier": "ZIGBEE_GW_RASPBEE",
    "fields": [
        {
            "name": "API Token",
            "key": "API_TOKEN",
            "_id": "6199783105707e5faa429369",
            "description": null,
            "value": "128ab9c6adb33c19878c0c67770618d3:74e4314d1083cfa5d9cc2550d97d1a6a:3de9d2c47a4e8e38e0886870d47c729ee994ed0fa4e24ba906180a954732d5a081a6ca57c08db289f51097c42f0fb992708fef34c561a92a2341bfd30763f2fd"
        },
        {
            "name": "Username",
            "key": "USERNAME",
            "description": null,
            "value": "acef9acddd0ad01a7e08ff39b5822da4:027a21c064b216e2287e705f5a72e9c1:88b1dd5acbf4af9a8c1f81bb2d6037d0afd7313ab7597b6b8eea359222a21151",
            "_id": "619e8169bf178d249985aab6"
        },
        {
            "name": "User Password",
            "key": "PASSWORD",
            "description": null,
            "value": "a2548cb348f783748b40509f51a1f525:66eee759d328be43cda38d30f2b30745:2bcdf96406ffa89d5dfa15106aaf0e8c",
            "_id": "619e815abf178d249985aab5"
        }
    ],
    "timestamps": {
        "created": 1637447729869,
        "updated": 1637783734049
    }
}

Update (2

{
    "_id": "6199783105707e5faa429368",
    "name": "ZigBee Gateway",
    "identifier": "ZIGBEE_GW_RASPBEE",
    "fields": [
        {
            "name": "API Token",
            "key": "API_TOKEN",
            "_id": "6199783105707e5faa429369",
            "description": null,
            "value": "128ab9c6adb33c19878c0c67770618d3:74e4314d1083cfa5d9cc2550d97d1a6a:3de9d2c47a4e8e38e0886870d47c729ee994ed0fa4e24ba906180a954732d5a081a6ca57c08db289f51097c42f0fb992708fef34c561a92a2341bfd30763f2fd"
        },
        {
            "name": "Username",
            "key": "USERNAME",
            "description": null,
            "value": "863f3b644f3986db4a85f7296e034688:b4c49cb12dc6c7f74c6d9a76e4ab59c2:f819cca65e8ca2ee68850bc4b665eb1a12c5c366a50ebc1d779cbc5e0ae447fa",
            "_id": "619e8169bf178d249985aab6"
        },
        {
            "name": "User Password",
            "key": "PASSWORD",
            "description": null,
            "value": "fb6178d9ad65875c15b7e5ed33900089:b850269177d2cfa555cc8ea0f399eec4:0e88af23a9a217df02cd2de51401c11c",
            "_id": "619e815abf178d249985aab5"
        }
    ],
    "timestamps": {
        "created": 1637447729869,
        "updated": 1637783734049
    }
}

helper/extend does not handle buffer objects properly

See unit tests, npm run test (dev branch)

      AssertionError [ERR_ASSERTION]: Expected values to be loosely deep-equal:

{
  arr: [
    6,
    7,
    8
  ],
  nested: {
    more: {
      final: {
        data: {
          '0': 76,
          '1': 111,
          '10': 32,
          '11': 104,
          '12': 101,
          '13': 114,
          '14': 101,
          '2': 111,
          '3': 107,
          '4': 115,
          '5': 32,
          '6': 110,
          '7': 105,
          '8': 99,
          '9': 101
        }
      }
    },
    string: 'Hello World'
  },
  obj: {
    prop2: false,
    prop3: 5000,
    prop4: false,
...

should loosely deep-equal

{
  arr: [
    6,
    7,
    8
  ],
  nested: {
    more: {
      final: {
        data: Buffer(15) [Uint8Array] [
          76,
          111,
          111,
          107,
          115,
          32,
          110,
          105,
          99,
          101,
          32,
          104,
          101,
          114,
          101
        ]
      }
    },
    string: 'Hello World'
  },
  obj: {
    prop2: false,
    prop3: 5000,
    prop4: false,
    prop5: null,
    prop6: 'string'
  },
  prop1: true
}
      + expected - actual

         ]
         "nested": {
           "more": {
             "final": {
      -        "data": {
      -          "0": 76
      -          "1": 111
      -          "2": 111
      -          "3": 107
      -          "4": 115
      -          "5": 32
      -          "6": 110
      -          "7": 105
      -          "8": 99
      -          "9": 101
      -          "10": 32
      -          "11": 104
      -          "12": 101
      -          "13": 114
      -          "14": 101
      -        }
      +        "data": [Buffer: [
      +          76
      +          111
      +          111
      +          107
      +          115
      +          32
      +          110
      +          105
      +          99
      +          101
      +          32
      +          104
      +          101
      +          114
      +          101
      +        ]]
             }
           }
           "string": "Hello World"
         }
      
      at Context.<anonymous> (tests/helper/test.extend.js:97:16)
      at processImmediate (internal/timers.js:456:21)

Logging output not shown in docker container

docker run backend openhaus/backend:latest & docker-compose up show now logger output on Terminal. Only the initial message from console.log is shown.

Test "Reminders": https://probot.github.io/apps/reminders/

/remind me To theck this tomorrow
/remind me To check this 10/12/2021

Cant update user again after one dataset update completed sucessful

Possible duplicate/related of/with #34

{
    "error": {
        "code": "ERR_VALIDATION",
        "details": {
            "_original": {
                "_id": "6110350990937c46ede24d98",
                "name": "Hans Werner - update 3525",
                "email": "[email protected]",
                "password": null,
                "timestamps": {
                    "created": 1628452104999,
                    "updated": 1634201781150
                },
                "enabled": true
            },
            "details": [
                {
                    "message": "\"password\" must be a string",
                    "path": [
                        "password"
                    ],
                    "type": "string.base",
                    "context": {
                        "label": "password",
                        "value": null,
                        "key": "password"
                    }
                }
            ],
            "isJoi": true,
            "name": "ValidationError"
        },
        "timestamp": 1634201781152,
        "message": "Validation on dataset failed"
    }
}

Remove update field completely from API calls and set/unset in component pre/post hooks?!