sonata-nfv/tng-tests

5GTANGO Integration tests repository

5GTANGO Integration Tests

This repository contains the integration tests as well as artifacts for the 5GTANGO components.

Folder Structure

Folder	Content
pipeline/	Jenkins pipeline scripts
packages/	Test NSDs, VNFDs, TSTDs to be packed to *.tgo packages
slas/	Examples of SLA Descriptors
tests/	Integration tests classified by SDK/SP/VnV
...	...

Artifacts

• 5GTANGO packages (*.tgo) build by Jenkins

Jenkins Integration

There is a Jenkins job to automatically build some of the artifacts stored in this repositry. For example, the Jenkins job automatically packages the SDK projects available in the packages/ folder. The resulting *.tgo files are then available as artifacts within Jenkins.

Licensing

For licensing issues, please check the Licence file.

Lead Developers

The following lead developers are responsible for this repository and have admin rights. They can, for example, merge pull requests.

• Felipe Vicens (felipevicens)
• Luis Hens (luishens01)

tng-tests's Issues

Slice - Share Network Service 3.2.2

The shared NSS is a CNF where the slices are (NS1(VNF) - NS2(CNF) - NS3(VNF)) and (NS1(CNF) - NS2(VNF) - NS3(CNF))

Task list:

• Create SP folder
• Create README
• Create .robot
• Create Jenkins job

Graylogs output into Robot

Add Graylogs error messages as an artifact if the Robot test fails.

Define devops workflow

Complete the information in the wikipage defining the DevOps workflow for the tests in each environment

test_sdk03_validation

Tool: tng-sdk-validate
Short description: Validate a project before packaging.
Parameters: Size (no. VNF), descriptor complexity (low/high), validation level (s,i,t), custom rules (number of rules)
Collected metric: time to validate, memory used by tool, (detection rate or similar? not sure if this works, to be decided by tool owner)
Priority: high
Complexity: medium
Responsible: Miguel, Ana (Quobis)

---

test_sdk07_vim-emu-scalability

Tool: vim-emu
Short description: Test scalability of vim-emu
Parameters: Number of emulated PoPs, number of deployed services
Collected metric: Boot times, memory, etc.
Priority: high
Complexity: low
Responsible: Manuel (UPB)

---

Note: Tests have already performed and results are available from previous Journal publication. We will just re-use those numbers for D6.3.

Simple Service Lifecycle - Multi VIM Network Service with 2 containers deployed in 2 VIMs

Task list:

• Create SP folder
• Create README
• Create .robot
• Create Jenkins job

Add package NSID1V_OSM

Simple Service Lifecycle - Multi VIM and the use of the WIM

Task list:

• Create SP folder
• Create README
• Create .robot
• Create Jenkins job

Slice - Share Network Service 3.2.1

Bothe slices are composed by 3 NSs, which they are all composed of VNFs.
NS1_1 --> shared_NS2 --> NS3_1
NS1_2 --> shared_NS2 --> NS3_1

Task list:

• Create SP folder
• Create README
• Create .robot
• Create Jenkins job

Simple Service Lifecycle - Multi VIM Network Service with 2 VMs deployed in 2 VIMs

Task list:

• Create SP folder
• Create README
• Create .robot
• Create Jenkins job

test_sdk08_benchmarker_performance

Tool: tng-sdk-benchmark
Short description: Benchmarking performance in terms of executed benchmarks per time
Parameters: Number of tested configurations
Collected metric: Reconfiguration time
Priority: high
Complexity: medium
Responsible: Manuel (UPB)

---

Data should be already available from collaboration with Ubitech. Just need to filter, polish and plot it.

Graylogs messages in Robot

Add Graylogs error messages as an artifact if the Robot test fails.

Add SLA test cases in:

https://github.com/sonata-nfv/tng-tests/wiki/test_cases

**test_sp01_test_integration_monitoring_vims**

Short description: Check the integration between monitoring manager and the VIMs
Tool: monitoring engine
Collected metric: test execution time
Priority: medium
Complexity: low
Responsible: PANOS K (SYN)

errors_sp_kpi

• endpoint and error are collected KPIs dimensions, so we can filter by it

request_latency_sp_kpi

• endpoint is one of the collected KPIs dimensions, so we can filter by it

Create a folder for the test in the route /test/SP/test_name

**test_sp05_test_migration_of_state (k8s)**

Short description: This test ensures that a state migration of a CNF is consistent.
Tool: MANO
Collected metric: test execution time, test instantiation time, test migration time
Priority: medium
Complexity: medium
Responsible: PANOS K (SYN), Eleni (UBI), Thomas (IMEC)

**test_sp04_test_migration_of_state (OS)**

Short description: This test ensures that a state migration of a VNF is consistent.
Tool: MANO
Collected metric: test execution time, test instantiation time, test migration time
Priority: medium
Complexity: medium
Responsible: PANOS K (SYN), Eleni (UBI), Thomas (IMEC)

client needs a testing_taged pair of a service and a test

Error when requesting map_package_on_service function from tnglib

robot test fails on map_package_on_service function
when called from /test_runtime_policy_lifecycle/test.robot file

This is the output as seen at command line after test execution:

==============================================================================
Test :: Test suite for enforcing a Runtime Policy to the SP platform          
==============================================================================
Setting the SP Path                                                   | PASS |
------------------------------------------------------------------------------
Upload the Package                                                    | FAIL |
KeyError: 'name'

Can we fix this? Am i requesting correctly the map_package_on_service funtion?
Thanks in advance :-)

Add package NSID3V (scalable NS)

Here I will need help!

Add package NSID2V

test_sdk01_project_management

Tool: tng-sdk-project
Short description: Create project with different sizes.
Parameters: Number of VNFs (1-1000)
Collected metric: time to create, memory used by tool
Priority: high
Complexity: low
Responsible: Stefan (UPB)

---

test_sdk05_test_packaging

Tool: tng-sdk-test
Short description: How long does test creation for VnV take?
Parameters: -
Collected metric: Packing time
Priority: high
Complexity: low
Responsible: Askhat (IMEC)

---

[Rate limit] Check rate limit responses when under the rate limit

1. Set the SP
2. Create a user with a role (eg. Dev)
3. Login
4. Get REQUESTS (2 reqs/1sec)
5. Check if 2 x 200 OK received (should be true)
6. Logout
7. Delete User

Create a file .robot with the definition of the sla test

Create a README.md with the test description

Add package NSID4VC: Mixed NS (CNF / VNF)

Create a new freestyle job in Jenkins for test 4.1

https://jenkins.sonata-nfv.eu/job/test-test_sla_e2e-4.1/

Add package NSID1V

name for disk image missing

Develop the Test Cases

test_sdk04_test_creation

Tool: tng-sdk-test
Short description: Test execution times (on-emulator)
Parameters: set of dummy tests
Collected metric: test execution time, service instantiation time
Priority: high
Complexity: medium
Responsible: Askhat (IMEC)

---

tng-tests documentation

Complete the devops testing methodology @felipevicens plus any additional information. There are some TST folders under the packages folder that are confusing. The wiki page is available.

Slice - Network Service Composition 3.1.2

Slice that is comprised of 3 NS, 2 with 1 VNFs and 1 with 1 CNF, linked among them

Task list:

• Create SP folder
• Create README
• Create .robot
• Create Jenkins job

Create a new freestyle job in Jenkins for test sla 4.3

**test_sp02_test_service_reconfiguration_(OS)**

Short description: Ensure that a service reconfiguration is triggered and executed correctly based on custom metrics from Openstack.
Tool: monitoring engine, MANO, Policy Manager
Collected metric: test execution time, test instantiation time, test scale time
Priority: high
Complexity: medium
Responsible: PANOS K (SYN), Eleni (UBI), Thomas (IMEC)

test_sdk02_packager

Tool: tng-sdk-packager
Short description: Package different packages.
Parameters: package size (no. VNF/files 1-100 VNF (maybe 1000)), package kind (5GTANGO, OSM), validation (yes/no)
Collected metric: time to package, memory used, package size
Priority: high
Complexity: low
Responsible: Manuel (UPB)

---

Simple Service Lifecycle - NSID1C1V Network service that deploys 1 container in one VIM and one VM in other VIM

Task list:

• Create SP folder
• Create README
• Create .robot
• Create Jenkins job

Create a new freestyle job in Jenkins for test sla 4.2

rate_sp_kpi

• endpoint is one of the collected KPIs dimensions, so we can filter by it

Slice - Network Service Composition 3.1.1

Slice that is comprised of 3 NS with 1 VNFs each, linked among them

Task list:

• Create SP folder
• Create README
• Create .robot
• Create Jenkins job

Unexpected field

The field slad is not in the schema sla schema

tng-tests/slas/silver_sla_example.yml

Line 6 in 6e09fe0

slad:

**

**test_sp03_test_service_reconfiguration_(k8s)**

Short description: Ensure that a service reconfiguration is triggered and executed correctly based on custom metrics from K8s.
Tool: monitoring engine, MANO, Policy Manager
Collected metric: test execution time, test instantiation time, test scale time
Priority: high
Complexity: medium
Responsible: PANOS K (SYN), Eleni (UBI), Thomas (IMEC)

Create a new freestyle job in Jenkins for test sla 4.4

Add package NSID1C

Manual scaling - 1 CNF 5.2

The manual scaling consists to scale out/in one CNF using the gk.
Task List;

• Create SP folder
• Create README
• Create .robot
• Create Jenkins job

Template .robot: https://github.com/sonata-nfv/tng-tests/wiki/methodology.md#test-case-descriptor
SP folder location: tng-test/SP/test-manual-scale-5-2

Slice -SLA within Network Slice 3.3.1

The Slice1 has the SLA1 applied, and the Slice2 has the SLA2 applied.
Task List;

• Create SP folder
• Create README
• Create .robot
• Create Jenkins job

Manual scaling - 1 VNF 5.1

The manual scaling consists to scale out/in one VNF using the gk.
Task List;

• Create SP folder
• Create README
• Create .robot
• Create Jenkins job

Template .robot: https://github.com/sonata-nfv/tng-tests/wiki/methodology.md#test-case-descriptor
SP folder location: tng-test/SP/test-manual-scale-5-1

Add package NSID2C