driver-eu/test-bed-admin

A test-bed administration management tool for managing the Kafka-based test-bed.

Admin tool

Single Page Application to help you manage your personalized DRIVER+ test-bed and prepare the trial. It uses Apache Kafka as well as an Embedded Tomcat environment. The frontend is built with Vuejs.

Introduction

The current version (v 1.2.29) of the Admin tool provides an overview of:

• the solutions connected to the Testbed, including their availability
• the available Kafka topics
• the gateways for message exchange

Furthermore, the Admin tool

• creates the necessary topics, assigns schemas to topics and handles topic invitations.
• provides the possibility to initialize the testbed
• provides the possibility to start the trial once the testbed was initialized
• provides a section where log entries are displayed
• provides the possibility to create solutions/topics/gateways

docker image

A docker image is on dockerhub available

• image: drivereu/test-bed-admin:latest

Following environment can be set:

• KAFKA_BROKER_URL --> string (e.g.: broker:9092)
• SCHEMA_REGISTRY_URL --> string(e.g.: http://schema_registry:3502)
• zookeeper_host --> string (e.g.: zookeeper)
• zookeeper_port --> int (e.g.: 3500)
• testbed_secure_mode --> 'DEVELOP', 'AUTHENTICATION' or 'AUTHENTICATION_AND_AUTHORIZATION'
• testbed_init_auto: 'false'
• reset_db: 'true'

And for AUTHENTICATION & AUTHORIZATION:

• management_ca_cert_path --> string (e.g.: http://localhost:9090)
• cert_handler_url --> string (e.g.: https://localhost:8443)
• cert_pem_handler_url --> string (e.g.: https://localhost:8443)
• security_rest_path_group --> string (e.g.: https://localhost:9443)
• security_rest_path_topic --> string (e.g.: https://localhost:9443)

Swagger

For testing the Admin tool offers also a swagger interface. This can be called by:

• http://host:port/swagger-ui.html

Configuration

The configuration is located in the config folder. Using the default configuration the complete testbed is running on the local machine. Changes for e.g. Kafka connection have to be made in the consumer/producer properties file. Details about the possible parameters can be found in the adapter description.

Solution configuration

Every solution that is part of the testbed configuration has to be added to the solutions.json file. The Admin tool loads the configuration on startup. As soon as a heartbeat (HB) message is received by the admin tool service, the solution will be marked as available and up. HB is checked every 5 seconds. If no HB is received within 10 seconds the solution is marked as down.

Topic configuration

All topics used for data exchanged have to be configured in the topics.json file. Each topic that is configured has to have an unique topic name. The core topics (type=core.topic) should not be modified as the adapters assume they are available. Standard topics (type=standard.topic) have to define

• which standard (by defining the msgType)
• in which version (by defining the msgTypeVersion)

Manual Configuration

The AdminTool offers also the possibility to add solutions/topics if they are missig in the configuration that is loaded on startup.

Add Solution

For adding a Solution click on the "CONFIGURE NEW SOLUTION" item on top of the Solution list. Fill in the data and cick "SUBMIT".

Add Topic, assign the schema and define publisher/subscriber solutions

For adding a Topic click on the "CONFIGURE NEW TOPIC" item on top of the Solution list. Fill in the data, select the schema/version and the publisher/subscriber and cick "SUBMIT".

possible msgTypes:

Corrently the AdminTool supports following msgTypes:

• cap
• largedata
• maplayer
• emsi
• mlp
• geojson
• geojson-sim
• geojson-photo
• named-geojson

Gateway configuration

Gateways are more or less solutions in the testbed but they have additional configuration properties and therefor they are handled seperately. Like solutions gateways send heartbeat messages and are marked as up or down in the Admin tool.

TTI Configurations

The TTI configuration can be created via json file that is loaded on admintool startup, or on the fly via the admintool gui. All configurations that are available can be displayed by clicking the Configurations button in the top menu.

For creating a new configuration, a click on the Create button opens the create configuration form.

Organisations

For grantig the access to the TTI and to topics a known organisation needs be be available. By clicking on the Organisations button, the list of current configured organisations is displayed.

Organistion can be edited and removed by clicking the corresponding icons.

For creating a new organisation, a click on the Create button opens the create organisation form.

Schema File Upload

If you want to use your own created schema or any other that is not supported, you have the possibility to upload the schema file so that you can register this to specific topics.

Initialize the testbed

After calling the Admin tool, it displays all information configured for the trial

• All involved solutions
• All used topics
• All gateways

Auto create mode

In the client.properties the parameter init.auto defines if the topic creation is done by invoking (either by clicking on the button, or invoking the Rest Endpoint by e.g.: Trial Manager), or if the Admin tool should create all core topics on startup automatically. Be aware, the Kafka has to be up and running before the Admin Service is started.

reset DB mode

In the client.properties the parameter reset.db defines, if the DB tables should be cleaned during startup. If this is set to true, all tables (also the logs) are cleaned and the configuration defined in the json files is stored.

Drow overview

To draw an overview picture how the solution are connected to the testbed and on which topics they are subscribe or are allowed to publish messages, press the Overview button in the menu. The image is opened in a seperate tab and can also be downloaded

Choose the active configuration

In the menu you can find a Configuration button. Pressing the button opens the menu with all availbable configurations. Selecting one will activate the configuration. Solutions, Topics and Gateways are automatically reloaded.

Choose Testbed mode

The default Testbed mode is Develop. Meaning the testbed is open, everybody can connect and register for message and can also publish messages. Topics are created on the fly when they are needed. Switching to Authentication means that only those solutions can connect that have a valid certificate to connect. Authentication and Authorization means, additionaly to connect via certificate also access to topics is granted by the admin tool. Only solutions that are allowed to publish or subscribe can publish or subscribe.

Create all topics

By pressing the "INITIALIZE TESTBED" button (right top in the header) the Admin tool creates all needed core topics and registers the corresponding schemas on them. The status of the topic creation can be seen in the log and by the indicator in the topic.

Start the trial

The trial is started by the user clicking the "START TRIAL" button. When the trial is started, all standard topics are going to be created, the schemas are registered and the invite message is sent to all solutions that are allowed to connect to the topics (as publisher or subscriber).

Solution availability

Each solution has an indication like the topics. As soon as the heartbeat from a solution is received the indication is set to green, indicating it is available.

Log list

Each log entry that is received by the Admin tool is stored in the DB and is visible on the Admin HMI.

test-bed-admin's Issues

Make frontend resource paths relative

For work on the test-bed compositions to make the services of the test-bed available on more user friendly URLs it will be possible to access the test-bed admin tool from a relative path, for example:

http://tb1.driver-testbed.eu/admin-tool/

However, currently the admin tool Frontend requests it resources using absolute pathing, for example requesting resources at:

'http://tb1.driver-testbed.eu/static/css/app.52e49e37aef43ba6d3600598ba1a1e3a.css'

This should be:

'http://tb1.driver-testbed.eu/admin-tool/static/css/app.52e49e37aef43ba6d3600598ba1a1e3a.css'

A fix should be as easy as removing a prefix '/' for the resource links.

Remove restriction for client id max length 25 chars

As the title says. Can we remove or increase the char limit for client id's? This is currently too error prone, because if a user of an adapter chooses an id over 25 chars the entire test-bed-admin will not work.

Show port number on console

When starting the admin tool, show the chosen port number on the console. We are using many ports, and it is difficult to remember what runs where.

Display active configuration and active mode

instead of displaying "Configuration" and "MODES" in the menu, display the active config and mode to see it without opening the menu.

Access to services via admin tool

Can the admin tool investigate the docker environment and add links to TMT, AAR, etc. E.g. when you open (expand) a solution, display a link so we can easily navigate from the admin tool to solutions/tools.

Create & download certificate

• For solution
  • Add create certificate & download certificate buttons
  • Endpoints:
    • GET /AdminService/createSolutionCertificate? clientID=<solution.clientID>&organisationName=<solution.orgName>&password=<solution.certPwd>&email=<solution.email>: returnValue is certificate file name
    • GET /AdminService/downloadCertificate/{fileName}: .p12 file

Add columns for simulators and test-bed services

The solutions column also contains the admin tool (bit strange, actually - shouldn't we hide it). Maybe better to add them to a separate column, services.

Message too long

Similar to issue number 13 but in the message field.

2018-04-26 10:01:03.253  WARN 6928 --- [      Thread-15] o.h.engine.jdbc.spi.SqlExceptionHelper   : SQL Error: 0, SQLState: 22001
2018-04-26 10:01:03.254 ERROR 6928 --- [      Thread-15] o.h.engine.jdbc.spi.SqlExceptionHelper   : ERROR: value too long for type character varying(2048)
2018-04-26 10:01:03.256  INFO 6928 --- [      Thread-15] o.h.e.j.b.internal.AbstractBatchImpl     : HHH000010: On release of batch it still contained JDBC statements
2018-04-26 10:01:03.264 ERROR 6928 --- [      Thread-15] e.d.a.s.controller.LogRESTController     : Error processing the Log message received!

Docker version

Please discuss with Pieter what is the best docker base image for the test-bed for all Java related services (also gateways), so we can limit the number of base images someone has to download when running the test-bed.

I don't want to have multiple Java base images. Probably an alpine Java version is most suitable...

Download messages for replay

Although the Kafka topics UI can be used to download messages, it is often not working properly. However, it would be great if we could filter the messages in the AAR and download them for replay in the Replay service.

Admin tool (and AAR) become unresponsive when there are too many log messages

For tb5, I've tried to initialize the Trial, but it fails (page becomes unresponsive), presumably because there are too many messages in the log. Same for the AAR.

Auto-discover services not working?

I can run the admin tool just fine, but it does not seem to pick up all services that are running? Do you automatically discover services, or are they predefined.

Add system_configuration topic to initialization of the test-bed

Document how to add a new schema to the admin tool

When a new schema needs to be added, document how to:

• update the java adapter
• update the admin tools
• publish a new version

Add TestbedMode dropdown

Add a dropdown to the menubar and display the current testbed mode.
The available testbed modes can be retrieved by HTTP GET: /AdminService/getAllTestbedModes

To select the current active mode you need to call: /AdminService/getActTestbedConfig
In this object the testbedMode propertie is the name of the current mode.
Select this entry in the dropdown

Add links to the applications (solutions, simulators, services)

It would be nice if, e.g. in the adapter's configuration message, it would contain a link to the actual service, so we could visit it from the admin's interface (open in new tab).

It would also require an update of the configuration message and adapter configuration (in order to share the link).

Edit button

• For solution, topic, gateway:
  • Add edit button left of status icon
  • Same form as for "New"
  • Endpoints

HTTP PUT
/AdminService/updateSolution
/AdminService/updateTopic
/AdminService/updateGateway

Sort solutions and topics alphabetically, so they are easier to find

And don't change place all the time.

Remove button

• For solution, topic, gateway:
  • Add remove button left of status icon
  • Endpoints:

HTTP DELETE:
/AdminService/removeSolution/{id}
/AdminService/removeTopic/{id}
/AdminService/removeGateway/{id}

Rename "Initialize trial" to "Initialize test-bed"

Since the trial initialization requires more, e.g. simulators need to be configured too. In addition, I would also like to use our test-bed for exercises, not only trials.

Provide feedback on UI for long-lasting operations

"Initialize test-bed" and "Start trial" button shall display indication as long as HTTP request is not finished.

create organisation handling popup

for managing organisation we need a popup in which we can manage:

• list the organisation
• edit an organisation
• create a new organisation
• remove an organisation

an organisation has following member
mandatory

• orgName
• userName
• userPwd
• certPwd
  optional:
• email
• phone
• city
• postcode
• street
  -nr
• description

change websocket library

We need to use socket.io library for WebSocket sommunication to the backend to run the backend behind a apache revers proxy

SSL reverse proxy support

With a web browser, if we try to access the web UI behind the Traefik reverse proxy on the testbed, we have a blank page because the webapp is not designed to work behind a reverse proxy. However, a reverse proxy is required for communication security, i.e. SSL/TLS protection. You can test it with an Apache or NGINX (with SSL enabled) instead of Traefik to reproduce the issue outside the testbed.

Filter support in log view

Provide filtering in the column, e.g. a multi-select to select which kinds of log sources, error or info etc you want to see.

Add configuration dropdown

Add a configuration dropdown in the menu bar.
The available configuration can be retrieved by a HTTP GET: /AdminService/getAllConfigurations
Use the name propertie in the dropdown as entry

To select the current active configuration you need to call: /AdminService/getActTestbedConfig
In this object the configName propertie is the name of the current active configuration.
Select this entry in the dropdown

Add system_timing_control topic to default initialization

Please, also initialize the system_timing_control topic by default. You can use the schema listed in the avro-schemas repo: https://github.com/DRIVER-EU/avro-schemas/blob/master/core/time/system_timing_control-value.avsc

Extend solution form

Extend solution form by:

• orgName: GET /AdminService/getAllOrganisations returns list of possible organisations for drop down
• username
• userPwd
• certPwd
• email

CRUD editing

New solutions and topics etc can be created, but not updated or removed.
Also, allow the user to create a new configuration via the GUI.

Critical errors do not show up in red

Do not show solutions that are not there

E.g. your FRQ COP tool is not available, so do not show it in the list.

Prepare client for multi-user support

The CORS handling in test-bed-admin needs to be modified in the same way as for after-action-review.

State comparison

When a trial (test-bed) is working fine, e.g. during a dry run, it would be nice if we could record the state (topics, services running etc) somehow, so we can later compare it to the actual state during the trial, and especially see the differences.

Add links to Landoop services (in the services column)

Link to Kafka Topics UI and Schema Registry.

Websocket connection fails on driver-testbed.eu

The admin-tool does not work when it is hosted on the driver-testbed.eu website. When opening the admin tool in a browser, it tries to make a websocket connection to ws://localhost:8090.

Filter using multi-select

Currently, when filtering messages, only one option can be selected, e.g. only show messages from the TMT. Preferably, it should be possible to make the dropdown filter a multi-select, so you can select messages from a number of sources, e.g. TMT and OST.

Testbed configuration overview

Add a button in the menu bar to display the testbed overview picture (SVG).
The picture can be retrieved by HTTP GET: /AdminService/createOverviewPicture

Display the picture in a separate tab and make is downloadable.