fair-acc/pulsed-power-ml

Studie Pulslast-Überwachung & KI-basierte Verbraucher-Klassifikation

pulsed-power-ml's Issues

Add the classification info to the data to be able to display the groups in the holoview dashboard.

[5pt,9pt] REST client with Emscripten

Fetch-API,
JSON deserilization with either IoSerialiser or other library that works with Emscripten

Tasks

• Implement fetch
• Set custom header
• Deserialize json

Findings

• X_OPENCMW_METHOD header missing for long polling

Implement algorithm for the calculation of the phase difference between voltage and current for a ~50Hz Signal.

[3pt,10pt] Visualize n Sinus-signals with m subscription in dashboard

Tasks

• Fix sample drops
• Implement multiple subscriptions
• list of subscriptions
• provide an additional query parameter with the last updated time stamp it received (e.g. lastTimeStamp=...)
• periodic polling (ideally in a separate browser thread)

Findings

• fair-acc/opencmw-cpp#231
• Emscripten Wasm Worker API fails to compile: emscripten-core/emscripten#17213

[8pt,16pt] GNURadio SINK to OpenCMW worker for n (Sinus-)signals

This shall be the final API,
Unit tests are included,
n signals x 5 time sampling factors need to be managed as the client may ask arbitrary combinations of signals.
Mapping between different signals and ring buffers must be implemented. If signals are not compatible or not existent, an intelligent exception must be thrown.

Tasks

• Implement for n=1 and single sample rate (part 1)
  • gr-time sink providing a callback function called from work
  • opencmw worker
    • RingBuffer stores chunks of data
      • tryPublish called from work callback
        • annotate chunks with current timestamp
      • periodic task calls poller and accumulates all chunks into Acquisition object and calls notify
  • unit tests with libhttp::client tests subscription to a sample source
• Implement for multiple sinks and multiple sampling rates (part 2)
  • gr-sinks register themselves at singleton registry with the signal name provided to the sink block
  • opencmw worker registers callback functions to all sinks and publishes their data via dedicated ring buffers
  • channel names consist among others of the signal name configured in the sink block and the sample rate, examples here: https://git.gsi.de/acc/specs/generic-daq/-/tree/DAQ_API#filters
  • Replace poller by "multi poller" which only advances all RingBuffers at the same time.

Findings

• MultiArray compilation error
• Subscription and query parameters matching
• httplib

Hyperparameter optimization

[3pt] Iteration on Look&Feel of the full dashboard

[5pt,5pt] Concept design and implementation for the integration of power consumption

Pre-integration in FEC (power consumption over the last 10 seconds).
Global integration in Prometheus.

Integration must work independent of FEC restart and independent of Prometheus polling rate.
It must also work if 2 collectors fetch data from different FECs.

[5pt,5pt] Full dashboard for operational visualization

[3pt,5pt] REST client for n signals, final time-domain API

Productive code with unit test coverage and full QA,
Packaging to manage long polling API,
No inspection of the DataSet interface

API definition: https://git.gsi.de/acc/specs/generic-daq#time-domain-fec-acquisition-properties-user-interface
use Annotated interface

Tasks

• Implement time-domain DAQ API object
• Deserialize MultiArrays
• Destride strided arrays
• REST client for n signals
• Visualization of n signals
• CI/CD

[3pt] Iteration on Look&Feel of the full dashboard for mobile devices

Create first draft of event / change detector.

Reading the data stream we want to detect if there has been a change and only then use the model to classify the new data.

Implement complete GNU Radio workflow w/o GNU Radio companion for the live application

Implement "calculate_background" function

[1pt,1pt] Procure and set up hardware test setup for 3 phases

Tasks

• Order hardware
• Hardware delivered
• Setup 3 phase setup

Create a first model to classify the incoming data into different devices.

[1pt,6pt] Counter visualization with ImPlot

Tasks

• Incorporate CMake
• Fix cast error

Findings

• Cross-Origin Request Blocked due to missing Acces-Control-Allow-Origin header in opencmw
• SyntaxError: JSON.parse: nlohman_json expects json to start with { and end with }

implement feature calculation functions

Create basic luigi tasks for training data creation and model training

[2pt] Initialize Prometheus DB, set up Prometheus without data reduction

Provide Readme / HowTo so that the set up facilitates for new contributors

[3pt,3pt] First practices with ImPlot

UI loop: data structures are put in a queue. At the beginning of the loop, the data structure of the queue must be transferred to concrete data structures for the visualization.

Tasks

• Implement queue
• First plots of self generated data

Implementation of model validation methods (plot functions, metrics)

[5pt] FEC worker interface defintion for Prometheus collector: Research

The Collector should be able to be reconfigured online without having to restart the Prometheus DB or the Prometheus Collectore itself.

[2pt] Simple engineering visualization using Grafana

The aim here is to be able to display the signals and read them out. The goal is to show that the data is available in the database.
The goal is not so much to reproduce the look and feel of the other dashboard. The goal of Grafana is to have a unit test for Prometheus - to check that the data is in the database.

[2pt,2pt] Frequency SINK to WORKER

Tasks

• detect if requested signals in one subscription have different sample rates
• Testcases for multiple channels/sampling rates

Pull Request: #100

[2pt] Switch GNURadio flowgraph to 3-phase-simulation

Tasks

• Migrate required pulsed-power GR blocks to pulsed-power-ml repository

Create holoview dashboard for stream data visualization

Create a holoview dashboard so that visualization is similar to the graph displayed in Ralph's presentation.

Code Example from holoviews gallery:
http://holoviews.org/gallery/apps/bokeh/streaming_psutil.html

pack trained model incl. feature calculation into ONNX container (to be used by inference tool)

[1pt,5pt] Visualize 1 Sinus-signal with 1 subscription in dashboard

Tasks

• Replace nlohmann deserializer by opencmw deserializer
  • Fetch ranges-v3 library in cmake when using clang
  • Add serialiser include target (cmake)
• Adapt nlohmann json deserialization to DAQ API
• Adapt buffer to DAQ API
• Adapt ImPlot::PlotLine
  • Format time axis to date/time
  • Convert unix timestamp to microseconds

Findings

• mp-units (required for opencmw build) requires range-v3 if it is compiled with clang < 14 which is the case with emscripten compiler (https://emscripten.org/docs/tools_reference/emcc.html).
• when using clang, dependency type resolving issues occur (during library includes, e.g. ambiguous symbols)
• ImPlot::PlotLine only supports homogeneous data types (e.g. int64_t/int64_t or float/float), but int64_t/float is not supported.
• ImPlot date/time axis formatting supports unix timestamps down to microsecond precision level using double epezent/implot#48 (comment). Timestamps in the time-domain DAQ API are given in nanosecond. For now, we will plot time in microsecond resolution as a max. 50 Hz sample rate results in 20 millisecond time intervals.

Implement database with average power consumption per known appliance

[3pt] Visualization for mobile devices

Implement "event detection" function

Create zmq stream pipeline with identity function for now and model plug-ins for later

Fix problems with Picoscope4000a block and GNU Radio 3.8+

[1pt,5pt] OpenCMW - Worker for counter

[2pt,2pt] Data reduction: Down-sampling (/60, /600)

Is to be executed with on-board means.

The downsampling is to be implemented in the collector.
This should actually only be a configuration of the collector.

Ideas

• each sink should provide multiple(e.g. 5) downsampled sampling frequencies https://github.com/fair-acc/gr-digitizers/blob/main/include/digitizers/cascade_sink.h#L21 https://github.com/fair-acc/gr-digitizers/blob/main/lib/block_aggregation_impl.h https://github.com/fair-acc/gr-digitizers/blob/main/lib/block_aggregation_impl.cc

Create 1st order labeled training data with the infoteam setup

Create a data set for each appliance which contains three phases respectively (off -> on -> off)

Each data set contains the measured voltage and current (and timestamp or index?) (raw data)

Training data should be stored as .h5 files on the infoteam NAS.

Implement gupta model (outer function / wrapper)

[3pt] FEC worker interface definition for Prometheus collector: Implementation

[8pt,5pt] Data reduction: Mean, Min, Max

This should be generic enough so that this data reduction also works for non-pulse load signals. In particular, this function should not depend directly on the variable names of the signals but work with "generic variable names".
This should be realized either also in the collector. Or in an adapter/plug-in between the collector and the DB.

Implement complete GNU Radio workflow w/ GNU Radio Companion for production of training data

Open Tasks:

• Configure GR that PQSPhi and FFT data is synchronized
• Test the whole flowgraph

Build test case to create live data

Add labels to training data

[2pt,2pt] Limiting curve as settings object in FEC

Create the worker with the Settings object.
The interface for the boundary curve should look the same as that of the time-domain API.

Pull Request: #104

Fit kNN classifier to labeled training data

Implement "subtract_emi_background" function

Add labels to measurement data

The measurement data will have no labels (i.e. active power per appliance), thus, such labels need to be added "manually".

For a measurement series with only one appliance being switched on and off 100% of the active power will be assigned to the respective appliance.

Once all appliances have been measured, we'll know how much power each appliance consumes, when it is switched of.

With the assumption, that the consumed power of all appliances is constant, it is possible to label measurement data with multiple appliances being switched on and off.

[3pt,3pt] First practices with GNURadio

Work through GNU Radio Tutorial

• Write own C++ block
• Write own SINK

Implement function to split data from measurements (with labels) into training data

The data from measurements will presumably have this format: [t, f_0, ..., f_n], where "t" is an index or a timestamp and f_0 to f_n are features (RMS_I, RMS_U, phase_shift, ...) evaluated at time "t", with one row per measurement.

After the labels have been added to the measurement data (see issue #XX) the format will be: [t, f_0, ..., f_n, p_0, ..., p_m], where p_0 through p_m is the current consumed active power consumed by an appliance at time "t".

The function to split the labeled measurement data will then split the data with a rolling-window of a fixed length "l" of timesteps and a stride of "s". One data point of the training data will consist of all values for all features from t_i to t_i+l (shape=[l, n]) and the respective label consists of the values p_0(t_i+l) to p_m(t_i+l), so the most recent distribution of the active power aver all appliances.
The next training data point will have all feature values and labels shifted in time by "s" steps (t_i -> t_i+s , t_i+l -> t_i+s+l).

[1pt,1pt] Extend visualization dashboard by frequency-domain signals

Pull Request: #86