The tng-industrial-pilot from sonata-nfv

Familiarize with Prometheus (and elasticsearch)

Basic idea and structure of Prometheus DB
How to query data from Prometheus DB
- With elasticsearch?
- Integration with Python, eg, via this package?
How to visualize query results? Any existing tools/dashboards that can be reused?

NS1: Setup SDK project for emulator prototype

Weekly testing [do not close]

This issue tracks the results of the weekly pilot tests performed by UPB.

VNF Edge Analytics Engine

Analyze machine data on a random basis, based on result execute immediate actions.

Leverage Grafana to build dashboards with nice graphs and alerts for the EAE: #43 (comment)

Build on existing Grafana docker image
K8s: Configure to connect to the CC's Prometheus CDU
K8s:Configure dashboard to show some useful graphs
K8s:Configure some example alerts with email notifications
vim-emu: Configure to connect to the CC's Prometheus CDU
vim-emu: dashboard to show some useful graphs
vim-emu: some example alerts with email notifications

Note: If we have many NS1 (vim-emu case) you will be always logged out if you switch from one Grafana browser window to another. Not a big deal, but don't be surprised.

Finalize VNF design and decide which third party components to be used.

See: https://git.cs.upb.de/5gtango/generalInfo/wikis/industry_pilot_vnf_cc_devpage

Setup Jenkins-based CI for this repository

We should have continues integration (automated testing) for this repo.

The tests should:

Build all Docker-based VNFs and FSMs/SSMs
Package (and thus validate) all SDK projects and store the resulting *.tgo files as artifacts
- ./pack.sh in sdk-projects

Probe implementation for VnV

Test Cases
We assume three test cases for the SM pilot:

Test case 1: NS1 tested in isolation

Input: Send 100 MQTT messages to NS1 (Probe A)
Tested: NS1 forwards MQTT
Output: Check if 100 MQTT messages are received (Probe B)

Test case 2: NS2 tested in isolation

Input: Send 100 Euromap63 messages to the Samba share of NS2 (Probe C)
Tested: NS2 translates Euromap63 to MQTT
Output: Check if 100 MQTT messages are received (Probe B)

Test case 3: Test NS1 + NS2 end-to end

Input: Send 100 Euromap63 messages to the Samba share of NS2 (Probe C)
Tested: NS2 translates Euromap63 to MQTT, NS2 sends to NS1, NS1 forwards MQTT
Output: Check if 100 MQTT messages are received (Probe B)

Probes Required

Probe A: MQTT generator

simple MQTT generator

Probe B: MQTT receiver

receive messages, log them and check if they are 100. Write results to shared folder.

Probe C: Euromap63 generator

This is basically the digital twin

FMP: Integration with service platform (instantiation of NS2)

FMP must be able to trigger the SP to instantiate NS2.
It is important to do this instantiation with the correct instantiation parameters, e.g., the IP address of the already running NS1 (normal) and NS1 (quarantine).

Factory control portal

UI for control of pilot demonstration. Manage machine park layout, add/delete machines.

Detailed definition of VNF features where missing

please ask if information is needed, might be topic for next telco

Fix crashing CC processor

When deploying the CC processor container through Kubernetes it repeatedly crashes. We should figure out why and fix it.

$ kubectl get pods
NAME                                           READY   STATUS    RESTARTS   AGE
ns1-cc-broker-deployment-867484cd6b-s8f7k      1/1     Running   0          8m5s
ns1-cc-processor-deployment-5d64659ff8-b7vnt   1/1     Running   3          8m5s
ns1-eae-deployment-787c944c87-8rcp5            1/1     Running   0          8m4s
ns2-dt-deployment-6f8486884-4x7cw              1/1     Running   0          8m1s
ns2-mdc-deployment-655dfddc7b-jxz6w            1/1     Running   0          8m1s

The logs are not very useful:

tango@fgcn-tango-k8s-2:~$ kubectl logs -f ns1-cc-processor-deployment-5d64659ff8-b7vnt
CC-CDU02 (processor): Starting Azure Cloud Connector ...

Any ideas? Maybe start by adjusting the print statements to use print(..., flush=True) to ensure output is flushed and shown in the logs for debugging?

Tags in tests / NS should not be the same

Please change the tags in corresponding NS / TD

tng-smpilot-ns1-emulator.eu.5gtango.0.1

@askmyhat @tsoenen

NS2: Setup SDK project for emulator prototype

NS
- VNFD for MDC (start with single CDU, empty ubuntu container)
- VNFD for digital twin (this is a shortcut, lets deploy the twin as part of the NS to get things running for F2F)
link artifacts from https://github.com/sonata-nfv/tng-industrial-pilot/wiki/Pilot-prototype-on-vim-emu
test running NS on vim-emu
document in wiki

FMP: Shown IP address is the one of elastic search but must be from "msf" container

seems like a state fetching problem

Digital twin - machine representation

First concept design created by WEID, to be discussed with UPB on Monday

Migrate Emulator Pilot to native K8s

This is an intermediate step before the pilot is deployed on K8s using 5GTANGO SP, we first want to have a version that is directly deployed on K8s without 5GTANGO.

TODOs

See corresponding wiki entry

1st working demo (for F2F in Dublin)

This issue captures the developments and TODOs for the first working demo we are going to show in Dublin.

Demo will involve: DigitalTwind -> MDC -> CC -> EAE all deployed on vim-emu.

Documentation: https://github.com/sonata-nfv/tng-industrial-pilot/wiki/Pilot-prototype-on-vim-emu

Todos and open issues

Deployment on vim-emu

VNF internals (software, configurations etc.)

Bugs

Running mosquitto blocks emulator shutdown (starting process in background with & solves it)

Others

create a detailed network diagram of the demo (show subnets, CDUs, IPs per CDU, OVSs, NAT, ...)
check if the old emulator dashboard still works (nice for demos)
- Not integrated anymore! Must do. Use temp solution on local machine (Bookmark bar "Emu. Dashbaord")

Known Issues

docker needs to run privileged to allow mount of SAMBA share (won't work in K8s)

UseCase #3: On-demand AR service

Provide relevant data for AR maintenance. One idea is to use data from VNF Edge Analytics Engine here.

VNF Intrusion detection system

Detect all attacks on machine network, created once per machine park

VNF Firewall

This VNF is used in several network services, specific paramter sets are needed.

FMP: FMP shows new NS2 as threaddetected, and it jumps back to isolated all the

seems like a state fetching issue

vim-emu: Pilot stops working correctly after about 5 minutes

When doing: https://github.com/sonata-nfv/tng-industrial-pilot/wiki/Pilot-prototype-on-vim-emu

The pilot stops working after about 5 minutes.

My first impression is that something between DT and MDC breaks and thus no new messages are published to the broker. The DT and MDC applications as such seem to be still running. Samba? (but this is only a first impression!)

I would assume similar things will happen on K8s.

Need to investigate further ...

SMP-CCS development

This small auxiliary component intents to solve the problem that FSMs are not reachable from the outside making it hard to trigger manual reconfigurations, e.g., from the FMP. The idea is to run this small component next to the FMP. The FMP could access it using a REST API provided by the SDMP-CC. The FSM in turn connects to the SMP-CCS and opens a gRPC server stream connection. Using the the SMP-CCS can trigger actions in the FSM whenever it wants, still the FSM needs not to be reachable from outside. The only assumption here is that the FSM can reach the SMP-CCS, which can easily be realised, e.g., by hosting the SMP-CCS in the public Internet.

Code location: tools/smp-ccs
README and documentation: here

The SMP-CC is deployed at UPB's premises and reachable from the Internet:

Host: fgcn-tango-smp-ctrl.cs.upb.de
gRPC: 9012/TCP
REST: 9011/TCP

curl -X GET fgcn-tango-smp-ctrl.cs.upb.de:9011/api/v1/ssmstatus

   {private}                        {public}
+------------+                 +-----------------+
|            |                 |     SMP-CC      |
|    FMP     |-----REST------->|(e.g. in Docker) |
|            |                 |                 |
+------------+                 +-----------------+
                                   ^        |
                                   |  (2) gRPC srv.
                                   |     stream
                               (1) gRPC     |
                                 req.       |
                                   |        |
                    + - - - - - - -|- - - - + - - - - - - - +
                                   |        v
                    |          +-----------------+          |
                               | +---------------+-+
                    |          +-+ +---------------+-+      |
                                 +-+       SSM       |
                    |              +-----------------+      |

                    |                                       |
                                 SONATA-NFV MANO
                    + - - - - - - - - - - - - - - - - - - - +
                                    {private}

Ports (e.g. fgcn-tango-vpn.cs.upb.de):

REST: 9011
gRPC: 9012

See: https://github.com/sonata-nfv/tng-industrial-pilot/wiki/FSM-SSM-Development

TODOs

vim-emu: Test deployment of multiple instances of NS2 all connected to a single NS1 instance

This should now be possible through: sonata-nfv/son-emu#294

This will be needed for NetSoft'19!

Create a second Wiki page to document the execution of such a scenario.

Challenge: OVS config for uplink port

FSM/SSM: Skip reconfiguration when state is already set (optional)

monitoring trigger
manual trigger

Not sure if this is really needed. But lets keep it in this issue.

SP-using-K8s deployment of pilot

Based on the native K8s services, and the emulator services, create a version of the pilot that can be deployed by the 5GTANGO SP on K8s.

See Felipe's initial work #75

See K8s-native issue #72

See updated Dockerfiles and descriptors: #77

Reconfiguration of MSF CNF with FSM

Once triggered, the FSM belonging to the MSF (MDC, Suricata, Filebeat) CNF should restart the CNF and set a new value for the MQTT_BROKER_HOST env var.

Will require adjusting the code of the FSM here: https://github.com/sonata-nfv/tng-industrial-pilot/blob/master/fsms/mdc-fsm/mdc/mdcfsm.py#L149

Belongs to use case 2: #31

VNF Machine data collector

Connects a machine to NS2. The machines provide an ethernet connector, exchange protocol has to be defined.

CC: Add Prometheus as edge database solution for sensor data to the CC

Prometheus seems to be a perfect fit for this sensor data use case at the edge.

Prometheus either pulls the metrics or needs a separated "pushgateway" to push data to it.

Design: Add a 4. CDU running the push Gateway. Let the CC processor push the metrics received on the broker to Prometheus.

vim-emu workflow

TODO

FSM/SSM: Update remote control server state when intrusion is triggered by e.g. monitoring

Will require a bi-directional gRPC streaming connection.

Create an empty folder structure for the VNF.

Set SMB client name to "machine1" in the IMMS

The IDS matches the SMB client names to detect bad clients.

To get this working, we need to set the name of the SMB client (the one running in the IMMS/DT container) to be machine1 (that is the default we agreed for now).

We need to investigate how this can be done. Need to google a bit.

IMMS_APP eats up 100% CPU

Not an issue in small deployments but becomes problem if many deployments are done.

Check if there is any busy waiting or so?

Setup empty development environment

NS2 IDS: Kibana server is not ready yet

Testing with the latest version of NS2+IDS, the Kibana server never comes up and trying to access it only shows Kibana server is not ready yet - even 10min after instantiation.

Example:

VNF Router

Connect and separate machine to machine park (NS1).

FMP: Integration with remote command and control server (trigger quarantine status)

The FMP must be able to trigger the remote CC server to trigger/reset the quarantine status of NS2.

This will be done through the REST interface of the CC.

The FMP should also display (and maybe constantly poll) the current quarantine status from the CC.

Cannot identify which machine sends the data if we run >1 NS2 connected to NS1

Notices this today:

If I run 2 instances of NS2 connected to NS1, NS1 receives MQTT messages like these from both NS2 instances:

WIMMS/EM63/DATE 20190613
WIMMS/EM63/TIME 12:34:54
WIMMS/EM63/@ActSimPara1 5
WIMMS/EM63/@ActSimPara2   1.3324
WIMMS/EM63/ActCntCyc 588
WIMMS/EM63/ActCntPrt 587

This does not allow to distinguish which machine sends which message.

So it will make sense to include the source machine name (whatever it is) into the topic, like:

WIMMS/EM63/ActCntPrt/<machine_name>

or

WIMMS/EM63/<machine_name>/ActCntPrt

Using subscribe commands with different wildcards, NS1 can then easily filter by machine etc. e.g.: subscribe WIMMS/EM63/ActTimCyc/+ gets a single metric for all machines. or subscribe WIMMS/EM63/+/m1 gets all metrics for machine m1 etc. etc.

Not sure if I will have time to work on this before the NetSoft demo. Let's see.

VNF Cloud Connector (CC)

UseCase #2: Isolate a machine

Control instance for the management of network services and VNFs based on machine topology defined in the portal.

sonata-nfv / tng-industrial-pilot Goto Github PK

tng-industrial-pilot's People

Contributors

Stargazers

Watchers

Forkers

tng-industrial-pilot's Issues

Test case 1: NS1 tested in isolation

Test case 2: NS2 tested in isolation

Test case 3: Test NS1 + NS2 end-to end

Probes Required

Probe A: MQTT generator

Probe B: MQTT receiver

Probe C: Euromap63 generator

TODOs

Todos and open issues

Deployment on vim-emu

VNF internals (software, configurations etc.)

Bugs

Others

Known Issues

TODOs

TODO

Recommend Projects

Recommend Topics

Recommend Org