vi-sense/vi-sense

vi-sense is a tool for visualizing 3d models together with IoT sensor data. This repo contains its backend written in go.

vi-sense

vi-sense is a project of the University of Applied Sciences (HTW) Berlin and metr.systems to visualize IoT data in BIM Models.

Vi-Sense is a project that visualizes data of heating systems in a 3D model of the building they’re installed in. Technicians or system administrators get an overview of all sensors located in a building, can take a closer look at the measured values of these sensors, and get information about data anomalies that indicate potential errors. The combination of the 3D representation of buildings and the visualization of measured values in a timeline makes it possible to perform detailed analyses of error sources such as the failure of heating or pump systems. In addition, sensor states in the building model are supported by color highlighting, which ensures a quick orientation and an intuitive overview. In short, the goal of Vi-Sense is to give technicians and building managers a clear overview of their properties and the sensor technology they use.

Installation

Local development

git large file storage is needed to install this repository

git lfs install

clone the visense-frontend repo

git lfs clone --recurse-submodules [email protected]:vi-sense/vi-sense.git

alternatively

git clone --recurse-submodules [email protected]:vi-sense/vi-sense.git

cd sample-data/

git lfs pull

or update repo

git submodule update --init --recursive

cd sample-data/

git lfs pull

Running

FRONTEND_DIR=[path_to_frontend_repo] docker-compose up --build builds and starts the app in dev mode, using the source code on the machine

FRONTEND_PORT=80 docker-compose -f docker-compose.yml -f docker-compose.integration.yml up -d starts the app in integration mode, using the visense image from docker hub, frontend listens on port 80

Generate API documentation

cd into app/ swag init -g api/api.go

vi-sense's Issues

Create project architecture

Create a UML diagram which shows the overall project structure.

analyse sensor data to show anomalies

• create backend functionality to analyze sensor data
• create an endpoint for this

As a user I want to be able to move and rotate a model in 3d.

Free movement in all directions and rotation around a single sensor.

document the API with swagger

Instead of documenting the api on our own we should use swagger to document it.

Babylon visual stuff

• import models and learn how to manage meshes and all their children

• change color/opacity of certain meshes on click or hover

Research on rendering and deployment of 3d models using three.js in the browser

For a first comparison a project should be created which loads the provided 3d models of metr with three.js. The application needs to be executable in the browser.

Furthermore, the performance of three.js in combination with WebGL should be researched.
Additionally, the availability of techniques such as occlusion culling and plane clipping should be examined.

As a frontend developer I want to access a single model by id from the api.

When the user selects a model a request should be send to the api which loads the required data (mesh, sensor data, sensor positions) from the server.

Research common error cases of heating systems

• what error cases can happen in a heating system
• which sensors are involved in those
• how can these error cases be visualized?

As a developer i want my code to be automatically deployed to a test server

Figure Out the details about persisting data when using docker container

• make the database persistent on the htw server
• store the 3d models in the file system of the htw server and make them accessable to the go container

Build our Babylon app as a module and import it in Vue SPA

Export babylon scene as a module
Import it into the Vue SPA

allow bidrectional communication between babylon and vue

• when a sensor is selected in 3d the sensor should also be highlighted in the information pane and vise versa

Getting started w/ docker

Research on the basics of how to use docker and a possible use in the project.

As a user I want to be able to select a single sensor in the model in 3d.

When the user selects the model, the user gets following information:

• Name of the sensor
• Description of the sensor
• (Child of the sensor)
• Temperature
• Time

See Mockup 2 Dasherexample: #8 (comment)

Install Docker on HTW Server

• install docker with apt-get or something like that
• enable starting the docker deamon on startup:
  sudo systemctl enable docker

create model page

additionally, create a model page to the already existing start page

• transform the start page into the model page or redirect

serve frontend with go

tests for db and endpoints

• right now there are only dummy tests implemented
• there need to be tests for every endpoint

As a user I want see a list of all the models to know which are available to view in 3d.

When the user opens the website a list of all avaiable models should be presented. When the user selects one model he will be redirected to the 3d viewer (described in #22).

Create minimal Go Server and Deploy with Docker

• minimal Go Server
• test for minimal Go Server
• set up automatic testing with GitHub Actions
• Dockerfile that builds runnable Server Docker image
• Dockerfile or Github Action that builds a minimal Server docker image (that contains only our binaries and is only 5mb)
• Deploy Docker container to HTW server with GitHub Actions

Provide more 3D models (of buildings)

Provide some more 3d models

update sensor

• add the ability to update sensor information

Create several test cases for testing the existing API

The whole API functionality with all endpoints should be tested to allow code stability even after refactoring.

As a user I want to be able to see all available sensors in the model in 3d.

When the user is entering the model in 3d, all sensors should be visible in the model and indicated with dots.

Add certificate to server and make it serve https

Merge Vue and Babylon Repository

As a developer i want to build and run my application locally (including services like the database)

As a frontend developer I want to access all available models as a list from the api.

get all request, gives basic informations like name, description, id

Contact university for a project server

• Research on specs
• size of models is important

Read about enviroment variables in JavaScript

Fix the firewall issue on the htw server

Docker doesn't work with the default firewall script that is provided by the htw engineers

Provide more data for errors

• pump failure
• temperature drops
• leakage

Get model files from Backend and load them in Babylon

Add a DB to the Server

Preferably using docker compose, so that the database runs in separate container

Research on rendering and deployment of 3d with an alternative framework

https://gist.github.com/dmnsgn/76878ba6903cf15789b712464875cfdc

For a first comparison a project should be created which loads the provided 3d models of metr. The application needs to be executable in the browser.

Furthermore, the performance of the selected framework in combination with WebGL should be researched.
Additionally, the availability of techniques such as occlusion culling and plane clipping should be examined.

visualize anomalies inside babylon

• explore stuff like shaders, opacity, scale, animations, colors, ...

Babylon technical Setup / Initialization

• Read about performance improvements
• setup TypeScript + Webpack
• install Babylon inspector
• Load glTF files properly: normalize/center the model, read about pivot/origin etc.

As a user I want to be able to select a single model to show it in 3d.

When the models get selected the necessary data should be downloaded to view it inside the 3d viewer (see #19). The user should get indicated that the model is loaded.

As a frontend developer I want to get information for a specific sensor.

sensor name, sensor type, unit of measurement, sensor data

build frontend github action

First concept proposal

Based on #8

Start Page

Requirements

• Some Information about what this application is, welcome Page
• List all available Buildings, Houses, Rooms, ... (Its not a model, its a real object)
• Select a building and go to Model Page

Optional Features

• Give information about building properties, type of building, its location, weather conditions at location
• Give basic status information (e.g. "All systems running well" or in red: "Danger! heating pipe exceed maximum temperature")
• Give basic status information about past events (e.g. "There was one alarm since your last visit")

Model Page

This is a distinguish page. It has to be visualized clearly! The page is built up from some basic components:

• Model pane (Babylon) for 3D model visualizations
• Information pane (Vue) for external 2D visualizations
• Timeline (Vue)
• Option pane (Vue)

We have two states

• No sensor selected (#State1)
• Sensor selected (#Satet2)

Basics of Model Pane and Information Pane

• Loading screen with progress bar for loading model files
• Sensor visualization, no sensor selected:
  • in Babylon: 3D model at sensor location or 2D overlay. Show very reduced information (e.g. sensor name, temperature. Only 1-2 information, not more!)
  • in Vue: (hierarchical) list of Sensors (collapsible)
• Sensor selection:
  • in Babylon: raycasting
  • in Vue: list item selection
• #State2, sensor is selected:
  • in Babylon: Make clear witch parts of the building/model the sensor belongs to (e.g. visually indicate its corresponding model parts as well, camera flight to sensor?)
  • in Vue: graphs of its data

Timeline and time selection

• when opening the page, the data shown in the graphs are up to date and in real time
• it is visually reduced in the beginning but can be enlarged
• the selection of a time has a direct impact on all(!) shown data!!! (Graphs, sensors, ...)
• #State2: show the senso data graph in the timeline

Option pane

• collapsible
• show state specific options
• not sure yet if really needed but i want all options to be separated from the information pane

Further ideas and notes

• Sensor data handling, how to find anomalies? We need to know their normal values (deviations, AI?)
• There is more then one data/graph for one sensor. So our graph visualisations needs to be scalble (e.g. multiple graphs per chart, multiple charts)
• Danger Mode: automatically select a sensor, when it has dangerous values

use string formatter in go

just a quick note, I'd recommend using string formatting with fmt
s := fmt.Sprintf("%s %s", ...)

vi-sense/app/db_handler.go

Line 226 in 9ed93a1

t, _ := time.Parse(Layout, line[0]+" "+line[1])

And test if the line actually has enough fields, what if there is only one char? The parser might fail to so just handle the error with a simple panic, since this is critical for the entire app.

Create at least three concepts for the User Interface with sketch

• Find out the minimal requirements
• Look at best-practice cases in the field
• Draw low-fi mockups with sketch (at least 3)

create information pane

• expandable list of all sensors (selectable)
• when expanded shows sensor information (name, description, current value, ...)

see #40

• Vuetify Implementation
• Sketch the whole Information Pane with all features by now
• Refactor Code
• CSS formatting (@dephiloper: need probably help on this step)
• TODOs onSensorSelectionChanged & getSelectedSensors in InformationPane.vue

Getting started w/ Go

Research on creating projects in go, working with go and deploying applications.

Research on rendering and deployment of 3d models using unity in the browser

For a first comparison a project should be created which loads the provided 3d models of metr in unity. The application needs to be executable in the browser.

Furthermore, the performance of Unity in combination with WebGL should be researched.
Additionally, the availability of techniques such as occlusion culling and plane clipping should be examined.

As a user I want see a list of all the models.

allow the access of model specific files instead of archives

Right now all model specific files are packed into one zip file. This should be changed to allow querying only the needed resources.

As a developer i want to be able to write tests that will be executed automatically

This includes unit tests as well as tests that require access to a test database

Provide list for mapping sensor IDs

provide list for mapping