hector6298/predictive_maintenance

real time telemetry readings, anomaly detection, routing on the cloud, and dashboard visualization

predictive_maintenance's Issues

Create IoT Edge module for machine learning inference

Is your feature request related to a problem? Please describe.
Create example IoT edge module for temperature and humidity. This feature is to get to know how the modules to send telemetry and inference would interact and does not focus on the actual ML model performance.

Describe the solution you'd like

• Code for IoT module on the Edge
• IoT Edge deployment for RPi
• Documentation on how the telemetry and inference interact and how to publish the messages.

Describe alternatives you've considered
telemetry and inference on the same module. However, we are going to prioritize modularity.

Additional context
The environment is ioT Edge runtime on Raspberry Pi. The modules are deployed as Docker containers on the RPi, the machine learning models will be trained and containerized using Azure Machine Learning API.

Design a framework to receive telemetry from multiple devices and present the information

Is your feature request related to a problem? Please describe.
Current dashboard implementation only displays information from a single device, we need to be able to support multiple devices.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
Instantiate a whole pipeline for each device, but that approach is undesired.

Additional context
None.

[Documentation] How to perform anomaly detection on real-time?

Is your feature request related to a problem? Please describe.
At some point features tables will be appended in real time to the database. As a PdM system, we need to be able to detect anomalies using statistical methods, before we can have an ML model.

Additional context

• Create Documentation on the wiki section on how to implement an anomaly detection system
• Include code samples if possible

[Documentation] Investigación acerca de microservicios para IoT industrial

Is your feature request related to a problem? Please describe.
Investigación acerca de microservicios para IoT industrial utilizando sistemas SCADA, PLC, etc. El objetivo de este issue es generar documentación para entender como podríamos descubrir los sensores y como enviar la data a IoT Hub.

Revisar el repositorio y la pagina

De manera Inicial:

• Se debería entender como usar el microservicio para descrubrimiento de dispositivos
• Entender como el servicio de publisher podría enviar data a azure IoT Hub.
• La investigación e implementacion de un microservicio para machine learning esta fuera del alcance de este issue. Se utilizará otro para tal objetivo.

Describe the solution you'd like
Documentación muy breve y presentaciones acerca del tema.

Additional context
El problema se divide en dos partes:

• Uso de IoT hub y microservicios industriales (Este issue se enfoca en este tópico)
• Servicios de azure para recepcion de telemetría

[Documentation] Servicios de Azure necesarios para mantenimiento predictivo

Is your feature request related to a problem? Please describe.
Investigación acerca de servicios en azure para captura y almacenamiento de datos proveniente de industrial IoT edge.
Procesos:

• Captura de datos de Azure IoT Hub
• Envio con Event Hubs
• Recepcion y almacenamiento con Azure Data Lake

Describe the solution you'd like

• Documentación muy breve y presentaciones acerca del tema.
• De ser posible elaborar un ejemplo

Additional context
El problema se divide en dos partes:

• Uso de IoT hub y microservicios industriales
• Servicios de azure para recepcion de telemetría (Este issue se enfoca en este tópico)

[Feature] Include rolling and tumbling features on telemetry data

Is your feature request related to a problem? Please describe.
This feature is to enrich the incoming data with rolling aggregates so that anomaly detection can be performed.

Describe the solution you'd like
Use Pyspark structured streaming and spark sql windows to compute rolling aggregates

Additional context
None

Buscar servicios requeridos para procesamiento en lotes

Add documents on wiki

When sidebar is collapsed, connection is closed on telemetry client

Describe the bug
When panel server is active and the sidebar is collapsed the telemetry client connection is closed.
Output when sidebar is collapsed is the following:

Message received:
    Data: <<{'messageCount': 36, 'humidity': 59, 'temperature': 28, 'timestamp': '2022-04-24\n16:47:15'}>>2022-04-25 11:31:11,494 Exception in callback <bound method BokehTornado._keep_alive of <bokeh.server.tornado.BokehTornado object at 0x000001B2FBE3C8E0>>
Traceback (most recent call last):
  File "C:\Users\mejia\AppData\Roaming\Python\Python39\site-packages\tornado\ioloop.py", line 905, in _run
    return self.callback()
  File "C:\ProgramData\Anaconda3\lib\site-packages\bokeh\server\tornado.py", line 760, in _keep_alive
    c.send_ping()
  File "C:\ProgramData\Anaconda3\lib\site-packages\bokeh\server\connection.py", line 93, in send_ping
    self._socket.ping(str(self._ping_count).encode("utf-8"))
  File "C:\Users\mejia\AppData\Roaming\Python\Python39\site-packages\tornado\websocket.py", line 445, in ping
    raise WebSocketClosedError()
tornado.websocket.WebSocketClosedError

2022-04-25 11:31:12,235 Failed sending message as connection was closed
2022-04-25 11:31:12,307 Failed sending message as connection was closed
2022-04-25 11:31:12,323 Failed sending message as connection was closed
2022-04-25 11:31:12,462 Failed sending message as connection was closed
2022-04-25 11:31:12,480 Failed sending message as connection was closed
2022-04-25 11:31:12,494 Failed sending message as connection was closed
2022-04-25 11:31:12,572 Failed sending message as connection was closed
2022-04-25 11:31:12,679 Failed sending message as connection was closed
2022-04-25 11:31:12,772 Failed sending message as connection was closed
2022-04-25 11:31:12,806 Failed sending message as connection was closed
2022-04-25 11:31:12,820 Failed sending message as connection was closed
2022-04-25 11:31:12,867 Failed sending message as connection was closed
2022-04-25 11:31:12,976 Failed sending message as connection was closed
2022-04-25 11:31:13,053 Failed sending message as connection was closed
2022-04-25 11:31:13,178 Failed sending message as connection was closed
2022-04-25 11:31:13,195 Failed sending message as connection was closed
2022-04-25 11:31:13,225 Failed sending message as connection was closed
2022-04-25 11:31:13,304 Failed sending message as connection was closed

Message received:
    Data: <<{'messageCount': 37, 'humidity': 58, 'temperature': 28, 'timestamp': '2022-04-24\n16:47:17'}>>
2022-04-25 11:31:13,476 Failed sending message as connection was closed
2022-04-25 11:31:13,506 Failed sending message as connection was closed
2022-04-25 11:31:13,521 WebSocket connection closed: code=1001, reason=None

Message received:
    Data: <<{'messageCount': 38, 'humidity': 58, 'temperature': 28, 'timestamp': '2022-04-24\n16:47:18'}>>

Message received:
    Data: <<{'messageCount': 7, 'humidity': 62, 'temperature': 28, 'timestamp': '2022-04-24\n16:46:37'}>>

Expected behavior
bar should collapse with no issues.

Add anomaly counter and present table of most recent anomalies on dashboard

Is your feature request related to a problem? Please describe.
In addition to the telemetry being displayed, we need to show the history of most recent anomalies detected in a table

Describe alternatives you've considered
None

Additional context
This history must also be saved on a data lake.

Create Azure IoT Hub service and explore device connection

[Documentation] Search for pricing of all suggested Azure Services

Describe the solution you'd like
Generate a report indicating pricing for each cloud service. Report should be on a spreadsheet.

[Documentation] Research on real-time dashboard technologies

Buscar servicios en Azure requeridos para procesamiento en tiempo real

Add documents on wiki

[Feature] Create a script for simulated IoT data

Is your feature request related to a problem? Please describe.
As no physical sensors are present at this time, we will need to simulate data generation and message pushing to IoT hub.

Additional context

• Create a local script for the generation and publishing of simulated data.
• Creation of IoT hub service and connection to simulated device.

[Documentation] Meeting con técnicos de cervecería

Is your feature request related to a problem? Please describe.
Reunio con técnicos encargados del servidor en cervecería.

Preguntas:

• Arquitectura y modelo de datos que maneja el servidor (OPC-UA, etc...)
• Prestaciones del servidor
• Flujo de los datos actual (preprocesamiento?)
• Factibilidad para implementar los microservicios en produccion en el servidor
  • Si no es posible, como podríamos interceptar los datos y enviarlos a otra máquina en tiempo real.
  • Si no es posible hacer capturas en tiempo real, es posible generar csv?
• Podriamos acceder a documentación existente respecto a la arquitectura actual y del servidor?
• Datos existentes de los sensores (telemetria)

Presentacion Cerveceria

Is your feature request related to a problem? Please describe.
No.

Describe the solution you'd like
Realizar presentacion de acuerdo a 5 topicos.

• Overview del proyecto
• Nuestra visión del proyecto y sus tecnologías
• Ventajas y desventajas de Cloud Computing vs Edge Computing
  - Costos
• Ventajas (No es necesario que vayamos)
• Nuestras preguntas para ellos
  • Como obtienen sus datos actualmente.
  • Como tener crédito en el proyecto.
  • Quien es que adquiere los servicios en la nube (Para crear Iot Hub y DataBricks-Credenciales de desarrolladores).
  • Permisos necesarios para empezar la primera fase.
  • Reuniones para mostrar avances

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Reunirnos 24/01/2022