xtuml/erebus

General purpose message driving Test Harness

erebus's Issues

/deployment/docker-compose.yml referencing old repo

Update docker-compose file to reference open source GitHub container registry

Add folder storing docker image into root directory

Update test harness process manager

• Process manager currently imports and uses full_pv_test from the protocol verifier package. This requires abstracting away.
• Check if protocol verifier is within the config file, if so run the tests.

Add bash script to pull and run protocol verifier

The end to end tests have the functionality to pull the latest stable release of protocol verifier, update necessary files, and run the test harness and protocol verifier with docker compose.

A bash script that automates this process for the user so they can quickly get up and running with the test harness and protocol verifier would be useful.

Outcomes:

Bash script written that pulls and runs latest stable release of protocol verifier

Improve Erebus README

The README that sits within the root directory requires uplifting and restructuring to improve its readability for new users to familiarise themselves and get started quickly with Erebus.

Outcomes:

• README updated and peer reviewed
• Test harness downloaded, built and used by third party using the information in the README

Update default_config.config to use Kafka config properties

Currently, the default config file uses HTTP as the default message bus protocol. For ease of use and testing with the protocol verifier, this should be switched to Kafka.

Outcomes:

• default_config.config file to be updated with Kafka config properties

Schedule the running of smoke tests each day

Description

To ensure environment availability schedule the running of all smoke tests each day at 4am.

Outcomes:

Jenkins updated to run the smoke tests each morning at 4am

Update PV kafka metrics ingest to use new JSON schema

The PV has been tagged at version 1.3.1-midtage3 Release testing CI after domain removal · xtuml/munin . This now uses another json structure for the collection of metrics from the benchmarking probe:

{"timestamp":"2024-05-23T14:23:06.980Z","payload":{"eventId":"7ff2da81-a94f-49b5-b9ef-c0febf16c981","tag":"aeordering_event_processed"}}

Outcomes:

Test Harness supports the above schema when ingesting metrics

Test Harness able to test 1.3.1-midstage3

Add standard doc folder to charon repository.

The repositories in the xtuml organization follow a convention with technical note documentation.
Set that up.

Move end-to-end-curl-commands.sh to scripts folder

File is currently living in the root directory.

Update end to end tests in CI pipeline with protocol verifier

Currently, there are no working end to end tests that test the test harness working with the protocol verifier.

Outcomes:

• Github actions to run and end to end test created on 1.3.1-mistage3
• Confirms run of functional test
• Confirms run of perfromance test

test-workflow.yml needs repo references updating to open source

Create Swagger docs

Using Swagger, create interactive API documentation

Update test harness reporting folder

• log_analyser is specific to the protocol verifier and can be moved into the pv package.
• report_results has protocol verifier specific code which can be abstracted away into the protocol verifier package

Github actions build-push.yml requires updating to opensource repo

build-push.yml file currently points at internal repo and uses SSH which is no longer required for the opensource repo

Update test harness requests_th folder

• request_logs and request_pv_io are pv specific and can be moved over to the protocol verifier package.

Update grok schema in line with new logging structure

Structure of logs have changed, eg from

2023-10-04T17:50:59.289109Z 1 svdc_job_success : JobId = b87dc318-b714-43ce-9ca0-0aac712f03e2 : JobName = test_uml_1

to

2024-05-31T13:20:17.698Z - {"jobId":"f67bd9b6-581a-4ca4-9fcd-d6eb4a6bd5f3","jobName":"simple_XOR_job","message":"happy success","tag":"svdc_job_success"}

Outcomes:

• log_analyser.py updated with new grok patterns
• Test files updated with new logging structure

Merge SmartDCSIT repository into erebus

• Merge SmartDCSIT test harness repository into erebus
• Update workflows to run only when in SmartDCSIT repository

Update READMEV2

Observations have been made on the new README that would make Erebus more accessible and usable, especially to those on Windows systems.

Outcomes:

• Update README/supporting code to ensure that the test harness can be built and run locally, on Mac/Linux/Windows
• Changes to the README are verified by a third party by building and running the test harness using the updated README.

Update test harness utils file

• Currently has a function mock_pv_http_interface which is pv specific and can be moved to the pv package.

Cache steps in Dockerfile

Dockerfile is currently written with this one RUN command

RUN chmod +x /test_harness_app/scripts/install_repositories.sh && \
    /test_harness_app/scripts/install_repositories.sh && \
    pip install -r requirements.txt

This can be split, so that the pip packages are cached first, as these are unlikely to change.

Outcomes:

• Faster time to rebuild as pip packages won't have to be redownloaded if they remain the same

Cleanup and tidy Erebus repo

General cleanup of repo following on from extracting protocol verifier logic out of test harness.

Dockerfile requires updating for open source

Current dockerfile references internal repos and uses SSH which is no longer necessary.

Refactor protocol verifier file structure

The protocol verifier folder currently has many files within it that can be organised into subfolders for a more logical file structure.

Add JSON parser

Add a JSON parser to the jenkins pipeline so that it will return a failure when invalid JSON is supplied using the 'additional info' box without having to run until the test harness returns a failure

Migrate protocol verifier tests

• Protocol verifier tests currently sit within tests/test_harness. These should be moved into the protocol verifier package under its own tests folder.

Repo is bloated with unused files/directories

Remove unused files/directories

Update test harness config folder

• test_harness/config/config.py has a class HarnessConfig that has methods specific to the protocol verifier. This logic will need to be moved to the protocol verifier folder.
• default_config.config has protocol verifier specific configuration. This should be moved into a separate pv specific header

Remove PVResultsDataframe class

This class is a subclass of PVPerformanceResults, however does not actually change any methods/attributes, therefore can be completely removed.

Outcomes:

• Class removed
• References to class updated to reference PVPerformanceResults instead

Docker compose not working on Windows

When running docker compose up --build on Windows, the following error is encountered "/test_harness_app/scripts/install_repositories.sh not found".

A temporary fix has been found that fixes this issue, adding the following to the Dockerfile

RUN apt-get update && \
    apt-get install -y dos2unix && \
    rm -rf /var/lib/apt/lists/*

and

RUN dos2unix /test_harness_app/scripts/install_repositories.sh

It is worth having separate Dockerfiles and Docker Compose files for both Windows and Mac/Linux

Outcomes:

• Dockerfile and docker-compose file created for Windows
• Functionality tested on Windows machine