optimistiksas/oibus

OIBus

OIBus

OIBus is a software application that can be run on Windows, Linux, and Mac operating systems. It is designed to extract data from industrial sources by utilizing multiple protocols, such as OPCUA-HA, OPCHDA, Modbus, MQTT and many more. It can also retrieve data by scanning folders. Once the data is collected, OIBus can transmit it to your enterprise applications.

Optimistik is using OIBus on many industrial sites to send data to its OIAnalytics solution to query from 10 to over 10.000 points with sampling rate at the second level.

OIBus can be installed and configured in minutes and does not need development skills.

Introduction

OIBus is intended to simplify the data collection. We, at Optimistik, felt we had a missing piece between NodeRed and proprietary products for a tool able to solve most of the common requirements for industrial communications and very fast to set up.

OIBus is composed of 3 layers.

• The Engine that orchestrates everything and is configured through an admin interface
• Several South connectors that will retrieve data from a given technology (SQL, OPCUA, MQTT, Modbus...)
• Several North connectors that will be able to transfer the data to application such as OIAnalytics, Rest API, Timeseries databases, MQTT broker...

You can learn more about OIBus by reading our documentation.

Build and deploy step

• Fork the OIBus repository and clone it. Be sure to have NodeJS and npm installed (LTS versions).
• Backend: open a terminal in the backend folder cd backend
  • Install the node dependencies: npm install
  • Run the backend: npm start
• Frontend: open a terminal in the frontend folder cd frontend
  • Install the node dependencies : npm install
  • Run the frontend: npm start
  • Access the application on http://localhost:2223 (default port)
• You can compile OIBus on your appropriate distribution. To do that:
  • In the frontend folder, run npm install and npm run build
  • In the backend folder, run the command associated to the distribution you want to build OIBus: npm run build:win, npm run build:linux, npm run build:macos, npm run build:macos-arm64
  • You can now start OIBus from its binaries with npm run start:win, npm run start:linux, npm run start:macos or npm run start:macos-arm64

A more complete developer guide is accessible on our developer documentation.

oibus's Issues

North Protocol - S3

Implement a north protocol that is capable of dealing with files and send them to an s3 bucket
Configuration of such protocol should be the following :

• Bucket name
• Path in bucket
• Access Key
• Secret Key

Make robust/test the grouping of multiple event from south into a single north update (to avoid latency issues with high frequencies)

Implement double control when modifying the configuration

The idea is to limit the number of reload and provide a way to double check configuration before it is actively loaded.

Split the oibus.json in two (active and current modification)
In the main page show the diff between the two if any. Have a reload button if there is any difference which will trigger a reload of the the new oibus.json config

Nice to have have a full history of all oibus.json applied

North applications should be able to "subscribe" to specific South drivers

We need to enhance fTbus so a North API can "subscribe" to selected instance of South instances.

Right now, every time a value is captured by a protocol, the engine adds the value to the bus and all North instances are called. We need a finer routage because some data (especially files) need to be routed to a given North API.

So we need to find:

• parameters to add into fTbus.json to describe the routage (the simpler the better)
• adapt the engine and other pieces to support this.

Current implementation requires 2 .node binary files to be present at runtime.

It's not a big issue but let's keep an eye on this.
2 ways I see to solve:

1. rewrite the cache with native nodejs component
2. wait until pkg comes with a solution...

No error handling when sending data to InfluxDB

When sending the request to InfluxDB with fetch the response status should be checked whether it was successful or not and manage the error

Limit IP adresses for koa server access (req.connection.remoteAddress)

https://stackoverflow.com/questions/28158333/how-can-i-get-the-client-ip-from-a-request-object-with-restify

better-sqlite3 problem after packaging

When I tried to see if the cluster solution works for restarting the app when fTbus is running as a Windows service I encountered the following error: WiseLibs/better-sqlite3#120

I tried some of the suggestions to reinstall node_modules and rebuild those binaries (better-sqlite3 and integer) but none of them solved the problem. It seems that something goes wrong during packaging.

I made a test with sqlite, which is just a wrapper to add Promise support to the SQLite3 package we used earlier, so we avoid having callbacks. It needs only 1 binary addon and, more importantly, it works. I was able to package the application and, with the native sqlite3 addon put near the executable, I was able to add as a Windows service and restart the app with the /reload endpoint.

Would you prefer using the sqlite package or would you like using a native JavaScript database implementation?

The switch to sqlite is already done, it took half an hour to replace better-sqlite3 (they have a very similar API). If you choose this solution I can commit and push it.

Logger is not able to log complex errors.

When trying to use logger.error(error), it just displays error: undefined, instead of logging the whole error with stack.
I think, it is because of how consoleFormat is defined, especially this part:
printf(info => ${info.level}: ${info.message})

When running the code, sometimes I see error: undefined, but I can't identify where is it coming from.

Create a north connector capable of handling raw file and send it through https : multipart POST

Principle

The idea is to be able to consume raw file from south application that emits them (see issue #75 ) and send them through a POST Multipart HTTP to an application capable of handling this raw file

The way the north application should work

• This north application should subscribe to south application sending raw files
• It should be configured the following way
  • Specify an url : (for example : https://demo.oianalytics.fr)
  • Specify an api endpoint : (for example : /api/optimistik/data/values/upload)
  • Specify an authentication type (for now Base64 authentication only should be supported)
    • For base 64 authentication (specify user/password)

Once configured it should upon reception of a raw file :

• Try to send the raw file (along with filename) using a Multipart POST (the file being one part)
• Read the response status (if != 200) -> buffer and retry later (see how this may be linked to buffering functionality)

Should the cache be "hidden" in the engine class.

Maybe the North could "acknowledge" if he was able to send or not the value to the end application. The Engine would be in charge to cache itself the value that were not properly sent and to resend them to the North application(s) on a regular basis.

Add a sql south application

Configuration

• Such south application should allow to connect to following databases by order of priority

  1. sql server
  2. mysql
  3. postgresql
  4. oracle

• It should have connection to db as parameters of the app (url, port, username, password, database type)

• It should allow to key in any number of sql queries and per query It should have a period parameter which controls how often the sql query is run

Behaviour

For each query specified and for each time period specified in the query the south app should run the sql query to the database (one db per south app) retrieve the resultset output it to a file and transmit the raw csv file obtained to any north app.

Wrong database name for InfluxDB

The database name for InfluxDB is not using the configuration for the InfluxDB application. Instead it is generated from the pointId received from the south driver.

Enhance raw file transfer south application

Sometimes it is not possible/suitable to move handled file from its original folder. Is it possible to enhance the raw file scanner south app to have three behaviour for handling a successfully sent file

• MOVE -> current behaviour with additional option of renaming or not and target folder configurable
• DELETE -> Delete file when successfully sent
• DO_NOTHING -> we leave the file in the folder

the latest option requires to maintain a list of file handled to avoid infinite loop. A modification to a file should trigger a resend

Set up simulation platform

Create a config file template with every possible north application and south driver as example

Create a South connector for raw file transmission

Principle

The idea behind this feature is to be able to transmit raw files to a north plugin capable of handling it. This is particularly useful when dealing with excel files or exotic csv formats.

The way it should work

Ftbus should allow configuration of this south connector with the following parameters

• A folder to scan
• Time period between each scan
• Regex expression to match the filename
• Minimum age of file to take into account (to avoid sending file in the process of being written)
• Behaviour with handled file
  • Keep in same folder : meaning a treated file should be kept in the same folder (and not treated again unless it has been modified)
  • Delete from folder : meaning a treated file should be deleted when successfully treated
  • Move to another folder (in this case we should specify a destination folder) : meaning a treated file should be moved to another folder the timestamp should be attached to avoid conflict with existing files

Once this is configured the south connector should poll at each scan period the different configured folders to look for files obeying the constraints provided in the configuration. It should transmit those raw files (along with the filename) to the north where only connectors able to handle such files can subscribe. Once successfully broadcaster it should implement the behaviour with the handled file.

It should be possible to define multiple south application to scan multiple folders

For S3 north app and Https multipart north app be able to go through a proxy

Sometimes at a client site it is necessary to be able to go through a proxy to reach internet. The S3 north protocol and the http post multipart should be able to specify a proxy setting in the form of :

• Proxy host
• Proxy port
• Username (optional)
• Password (optional)
• Certificate?

Test/make Robust the local cache if communication is cut with north

Hash the password of admin in the ftbus.json file

https://stackoverflow.com/questions/17201450/salt-and-hash-password-in-nodejs-w-crypto

Fix duplicated modbus/Modbus folders

The south folder contains 2 modbus/Modbus folders. The difference is that the first letter is uppercase or lowercase. Probably they were created on a non case-sensitive file system and it messed up them.
You can check it also by browsing the source code on the GitHub web interface.

Finish web support to edit fTbus.json remotely

Please review the PR. It's quite a large one as I had:

• remove webpack on the server side (as pkg is doing the job to mix the server in a single executable)
• create a webpack config for the client (with React as engine)
• implement a proto using React-Json-form that allows to create form automatically based on the schema definition
• used bootstrap as CSS framework (to be consistent with OIAnalytics)

After this:

• we will need to finish the submit (need a router.post('/config') to update the ftbus.json file
• extend the forms to North/South and other sections
• add as much as possible hint/help in the form so user don't have to read a doc

Should we use nodejs eventemitter in the communication north/south

from https://www.joyent.com/node-js/production/design

const EventEmitter = require('events').EventEmitter;

class MyClass extends EventEmitter {
constructor() {
super();

setTimeout(() => {
  this.emit('myEvent', 'hello world', 42);
}, 1000);

}
}

Inconsistent addValue() call

Different south protocols calls the Engin's addValue method with different parameter set and type:

• Modbus and OPCUA are using an additional "dataId" field which is not handled at all by the Engine
• MQTT sends "data" as String, Modbus sends "data" as Boolean or Number, CSV sends "data" as Object and OPCUA sends "data" as Array

Is JSON.parse() in InfluxDB.class.js and TimescaleDB.class.js properly handling all those possible types?

General code review (style/doc/architecture etc...)

Create a deployable executable on windows for fTbus

We would like to start deploying fTbus on customer infrastructure. For that we need "something" that is easily deployable.

We want to focus first on windows os deployment.

The idea is to have everything bundled in one file and that can be registered and started easily as a windows service.
Is that achievable?

Create a localhost web page that permits the configuration of ftBus

This web interface should allow :

• Adding/editing a south application
  • For each south connector being able to configure its characteristics
• Adding/editing a north application
  • For each north connector registerd being able to configure its characteristics
• Configuring the engine charecteristics

It should support internationalization
It should support form validation
It should support hot reload upon modification

Set up ESDoc and Jest

Use fTmotion settings

Refactor the web-service implementation

InfluxDB requires timestamp to be nanosecond-precision Unix time

Delete application/equipment with confirmation

OPCDA south plugin (or Beckhoff?)

The current cron package does not go below second resolution so we should use timexe

https://github.com/paragi/timexe#readme

**await fetch()** shouldn't be in a try-catch? in the client services

don't you have to check if response.ok is true when using fetch? The documentation clearly states that it will resolve normally even if the response is HTTP 404 or 500. With a custom 404 handler on server (not the case in our implementation) you can receive a valid JSON response.

TimescaleDB north plugin

Profinet South Protocol (or Beckhoff?)

this variable may return an error and this need to be managed

https://github.com/factoryThings/fTbus/blob/eb9b7542b55e1e708de2226e7e4f0b521a7524a2/src/engine/Engine.class.js#L167

Implement a more thorough config file validator

North Application - Alive signal

Add a north application that will be used to send alive signal to a remote host. The following configuration elements should be made available for this app :

• Remote url
• Proxy (optional if access to remote is done through a proxy)
• Oibus friendly name
• Frequency

At frequency rate the url should perform a get of the remote url with a request parameter id equals to oibus friendly name

Use Logger instead of console.log/error/info/debug

rename ftbus to oibus

due to rebranding we are renaming ftbus to oibus. Can you proceed to renaming wherever ftbus appears it should now be oibus.

When a marketing name appears it should be spelled OIBus otherwise lowercase oibus is good for technical naming

Fix problems reported by ESLint

Everybody should check the code with "npm run lint" before every commit.

Add unit test with jest

Please provide one PR for each unit test file.

Command line arguments

In config.service.js the isValidArgs() function is kinda useless, because the config parameter has a default value specified, so it won't be null or undefined.

If we will distribute the application as an executable it would be useful to add some additional command line arguments, like:

• --version: to easily get the version (using the "version" field from package.json maybe?)
• --help: to list and document the available command line arguments

If we plan (and we plan, because there is an issue for it :}) to make the application configurable through the built-in web-server it may be a good idea to add some additional command line arguments, like:

• --port: the port for the web-server to listen on (can be optional and fall back to 2223)
• --user: the default admin username (can be optional and fall back to "admin")
• --password: the default admin password (can be optional and fall back to "admin")

This way the client simply starts the app with "./fTbus"/"fTbus.exe" and access the web interface for additional configuration. No need to manually write the "fTbus.json" config file.

Check the connection process with OPCUA

const client = new opcua.OPCUAClient();
await client.connect(endpointurl);

if (client.isConnected) {
console.log("client is connected");
} else {
console.log("client may be trying to reconnected");
}
alternatively , you can listen to some event that are raised when connection is changing

client.on("connection_reestablished",function() {
});
client.on("backoff", function (retryCount, delay) {
});
client.on("start_reconnection", function () {
});

in next version there will be also:

    client.on("connection_lost", function() {
     });

Complete .gitignore

• Generated folders, like docs and dist should be gitignored
• Logs folder should also be gitignored and journal.log should be stored inside this folder
• Decide whether to include IDE's config folder. E.g. I'm using IntelliJ Idea which stored it's project related configurations inside ".idea" folder. This list also includes the list of opened files, the project structure tree expand/colleps status and also the cursor position inside the files. So it definitly should be gitignored.

Fix ESDoc problems

• there are undocumented functions (can be forced with "valid-jsdoc")
• there are undocumented function arguments
• there are partially documented function arguments - mentioned, but not described what they actually represent
• there are documented function arguments which are no longer present as function arguments
• Object arguments should be further documented (like myObjectVariable.field1)
• there are undocumented return values
• misdocumented @extends directives

CSV.class.js uses this.engine.bus.emit() instead of this.engine.addValue()

With the new UI, we dont need to check here again. So we should simplify.

https://github.com/factoryThings/OIBus/blob/b48243a69b5d6fa6fcae386412617b48d575bdb6/src/engine/Engine.class.js#L28

Disconnect from TimescaleDB after the query is sent

It's rather a question than a bug report.

If we initiate a new client and create a new connection to TimescaleDB on every onUpdate call, don't we have to end the connection after the query is sent?