santoshdhirwani/benchmark_ethereum

This repository was created to automate the benchmarking process of private ethereum deployment with caliper.

benchmark_ethereum's Issues

Adding a flag in main.py to ask user about printing output

Task

Adding a flag in main.py to ask the user whether he/she wants the detailed output to be printed or not.

Detailed Description

At present, a lot of things are being printed while running the tool which makes it very difficult for the user to understand exactly which step is being executed.
We need a flag in main.py for the user to specify if the detailed output is required to be printed or not (beneficial for debugging).
This task should add meaningful comments in main.py to print clearly which step is being executed at the moment.

Acceptance Criteria

• The code must run without any errors.

• The execution must print clearly the steps of the execution and results of each step in case the flag is enabled for used mode.

• The execution must print all the output (as it is now) in case the flag is enabled for debug mode

Create a first draft for the final presentation

Task

Detailed Description

We need to create a first draft for the final presentation so we can get some early feedback from Jacob in the next meeting (04.02.2020).

Notes

Acceptance Criteria

• The presentation must have a structure

• It must have all content which is more or less ready, for example, all of the theoretical content, system design, etc.-ON HOLD due to results

README and repo organization

Task

Detailed Description

The repo is accumulating scripts and the README needs to be updated accordingly.
The README should include not only the directory map of the repo but also the installation and execution instructions. It must also explain every value in the json config file and list every dependency used.
The repo structure should be reorganized to be more descriptive, specifically follow the modular organization.
Inside bin we should have the following:
main.py
sut/ - with all the SUT deployment automation scripts and files needed
workload/ - with all the workload (caliper) scripts and configuration files needed
analyzer/ - with all the aggregator scripts and possible scripts that deal with reading results.

This organization needs to change correctly every path that is written in every script or errors will appear.

Acceptance Criteria

• The tool is running without any errors

• The README is clear and describes all the information requested in the task

Associated story

#30

Notes

#98

Automated genesis improvements

Task

Detailed Description

We need to receive nodes, gas limit and block interval as argument and create the same number of accounts as nodes locally with geth.
Like in task #18 It will create the genesis file taking block interval and gas limit as well as the accounts that should be added as sealers. This must be done without using puppeth

Acceptance Criteria

• The code of the script runs without any errors (error handling correctly used).
• The script receives via arguments the number of nodes, gas limit and block interval.
• The script must create a new genesis file with same number of accounts as nodes and the specified gas limit and block interval.
• The genesis file and they keystores created should be tested, creating a network that works with geth. Any account must be able to seal.

Associated story

#18 #54 #13

Notes

Change installation method for Caliper and add instalation for web3 in prerequisites.sh

Task

Detailed Description

Currently our prerequisites.sh file installing Caliper using Global NPM method. However this method not working well in practical. We need to change using Local NPM method and put Caliper inside folder workload.

Another package that need to install is web3. This web3 should installed inside workload folder too.

Associated story

Notes

Improvement from current prerequisites.sh file

Create main user script

Task

Detailed Description

We need to create the main script that the user will run from the CLI. This script will trigger the other scripts we are implementing.

Associated story

#13

Subtasks

#38

Notes

This task is still in progress, subtasks will be made to implement this script.

Write issues with detailed description for the upcoming sprint after planning with team.

The issue must include User Story, Tasks, Acceptance Criteria, Notes, Deadline and Story Points.

GCP Instance template creation

Task

Detailed Description

#33 MUST BE FINISHED BEFORE TAKING THIS TASK
We should create a script that creates a VM template. The process should be the following:

• Create a VM
• Deploy the script #33 and execute it.
• Stop the VM
• Create a VM template based on the VM created in the previous steps
  This script should read from the config file the name of the template given by the user

Acceptance Criteria

• The code of the script runs without any errors (error handling correctly used)
• The script creates in the configured project a VM template with the configured name
• Test if any VMs created from the instance template created by the script are able to execute geth and puppeth

Associated story

#33 #13

Notes

Gas limit addition to genesis file in SUT script

Task

Detailed Description

The SUT script uploaded #47 works with some considerations to further develop. One of them is changing the gas limit in the genesis file.
After creating the genesis file, we need to edit the json file and write the gas limit received as parameter in hex value.

Associated story

#13

Notes

Create Workload deployment script

Task

Detailed Description

We need to create a workload deployment script. This deployment script will use the caliper image and will build the same number of containers as number of nodes of the SUT. When deployed, each container will benchmark one node and generate results.

Associated story

Story: #22 Workload deployment

Notes

We need to discuss how are we going to gather the experiment result from each container

[Subtask] Implement lower bound gas limit algorithm

Task

Detailed Description

We need to implement the following algorithm.
https://github.com/Cloud-Prototyping-WS-2019-20/cp_ws_1920/blob/algorithm-flowsharts/documentation/Lower%20Bound%20Gas%20Limit.pdf

Acceptance Criteria

• The code of main.py executes without errors.

• The implemented code executes correctly the algorithm defined and obtains the lower bound block gas limit

Associated story

#76 #77

Notes

Implement feature to build the SUT nodes in different zones / regions

Task

Detailed Description

We need to change deploy-sut.sh to build nodes in different zones / regions if the user asks for.

Acceptance Criteria

• The code runs without errors

• The SUT builds nodes in different regions / zones

Associated story

Notes

testing connection to slack

Test

Mockup script to find the optimal value of block gas limit and block interval.

Task

Detailed Description

We need a script that reads the results of the experiment and finds the optimal values.
Specifically this mockup script can read for now from a file fake experiment results (block gas limit, block interval, throughput) and it must find minimum the block gas limit and block interval values that offer the maximum improvement.
The script also receives the sensitivity configuration parameter (minimum difference value of improvement to keep searching for optimal values).
Find the mathematical value for parameters value to test, by researching about operational research

Associated story

Story: #14 Block gas limit and block interval optimal values

Notes

This story can benefit from the results of #19

Modify Script aggregate-results to create a meaningful report

Task

Detailed Description

Right now main.py script is giving the final result in a print command. We need to generate a meaningful report to the final user where he can see the final output of our tool and meaningful insights of our execution.
The idea is to modify aggregate-results so that it creates a HTML format report with the information of our execution, plotting useful graphs.

Aceptance criteria

• The script must run without errors

• Once the execution of the tool is finished a report in HTML should be available to open and see the result and meaningful insights of our execution.

Associated story

#14

Notes

SUT Script Improvements

Task

Detailed Description

Make the the network id, the username and the password configurable in the config file so it can be read by SUT-deploy script.

Right now INSTANCE_LIST and INSTANCE_IP_LIST are extracting different information from the same list and we are assuming that the output will list in the same order. To avoid having future problems with our assumption INSTANCE_LIST and INSTANCE_IP_LIST should be merged into one. it will also need changes in the gcloud command and script adaptation.

Acceptance criteria

• The code of the script runs without any errors (error handling correctly used).
• The script receives as arguments the variables specified in the task.
• The script gives the same result even with the changes applied.

Associated story

#13

Notes

Prereq update

Task

Description

Add to the prerequisite script the installation of pm2, mention the right versions of packages and document correctly the needed version is the README.

Acceptance criteria

• The code must run without errors.

• The information should be self explanative and clear for the user

Notes

Script to run caliper to benchmark the blockchain

---

due: 2020-01-12

Task

Detailed Description

We need a script that runs the caliper command to benchmark the blockchain. The script should:

1. Receive as exec parameters the IP and port of a node, the account and its password.

2. Write under the ethereum network configuration file of caliper the configuration needed to benchmark a node with the parameters received

3. Execute the caliper command to benchmark.
   We will run caliper locally, we will not use the docker image.

Associated story

#22

Notes

Block gas limit and block interval optimal values

Story

User story

As a user, I want the analyzer to give me the optimal ethereum block gas limit and block interval

Tasks

• #19 Find useful literature that research about block gas limit and block interval impact.

• #20 Mockup script to test the optimization of block gas limit and block interval.

• #21 Script to aggregate the results of the tests in one place

Acceptance criteria

• The Analyzer should give me as an output the optimal block gas limit and block interval

Notes

Script to aggregate the results of the tests in one place

Task

Detailed Description

We need a script that aggregates the HTML results of the tests.
Caliper generates HTML reports from the experiments, in order to treat the information we need, we have to go through the HTML reports and gather the values we need.

Associated story

Story: #14 Block gas limit and block interval optimal values

Notes

This task is open for discussion, we still need to know the values that are going to be read (we may have several contract functions to read). We also need to agree if either this script will send the results to other script or if it should write them in a file that later on will be read.

Project Documentation

Story

User story

As an end user, I want to have the documentation about the tool and research done to build it

Tasks

• Final report first part

• Requirements to build and run the project

• Readme with procedure to run the tool

Acceptance criteria

• Final report finished

• Final presentation finished

• Tool documentation finished

Notes

Monitor the network stats for combination of gas limit and block interval

Task

Detailed Description

We need to experiment and see what is the behavior of the network for different gas limit and block interval configurations, we are interested in seeing what can be the normal stats when executing caliper and what are the stats when executing caliper with a extreme gas limit and block interval configuration.

We also need to know if there is a better way to detect a possible stagnation or benchmark crash than just executing caliper (e.g. block propagation, latency). In order to find out which values could be important and how to actually monitor them in our tool, we need first to analyze with an existing tool different benchmark situations.

Using something like https://ethstats.net/ can be interesting for this task.
https://medium.com/coinmonks/how-to-build-ethereum-dashboard-and-to-monitor-your-ethereum-network-status-9f1941beac08
https://github.com/goerli/ethstats-server
https://github.com/goerli/ethstats-client

Acceptance Criteria

• Find out what is the expected behavior of the network for the workload it is configured now in our tool

• Recommend stats that we can use to detect a possible crash or stagnation

Associated story

#14

Notes

Add project management techniques and task split up to midterm presentation slides

Modify aggregate-reports script to create a file with a grid of results.

Task

Detailed Description

In order to make easier the implementation of calculate-optimal-values.py added in #57, we need to modify aggregate-html-reports.py to generate file with a grid of the results.
This script should receive as parameters the maximum and minimum values of both block interval and block gas limit and their steps. Then for each function in the workload benchmarked (for now open and transfer) it should create a file with a grid of results. This grid should have same number of rows and columns as block gas limit and block interval respectively, separated in comas and line jumps.
Results must be organized correctly in the grid. Imagine 3x3 example:
min gas limit = 10m, max gas limit = 20m, gas limit step = 5m
min interval = 10, max interval = 20, interval step = 5
The result is organized this way (x= gas limit, y= interval)
.........10,15,20
10m 15,17,19
15m 25,27,29
20m 30,32,34

Being the file output this format:
15,17,19
25,27,29
30,32,34

Associated story

#14 #57

Notes

Research how to dynamically public miner nodes services in the same cluster

Task

Detailed Description

The idea is to use a script that uses Google SDK to connect to GKE and manage the SUT. We need to know if, after deploying with Kubernetes the SUT in the same cluster, we can receive an event warning us the the nodes have been correctly deployed. This will be needed to know when to accept a new sealer node in the network.
GKE will create containers in a cluster, the idea is that we will have one node per container. Theoretically the nodes will be built in the same machine, which means that these can not use the same ports. However with GKE we have the possibility of publishing services to access containers that can have different IP:Ports addresses even if they are in the same cluster/machine. We need to know if the use of services can solve our problem of publishing the Ethereum nodes even if these are in the same cluster.

Associated story

Story: #13 Ethereum SUT Deployment

Notes

The research should be documented commenting in this issue with the sources found. If possible, suggest a way of applying the solution.

Modification of the result aggregating script to take into account rounds

---

due: 2020-01-10

Task

Detailed Description

We can actually run several rounds in Caliper. The idea is that the aggregating script should also gather these rounds from the report files.

Associated story

Notes

Run-caliper script error handling

Task

Detailed Description

We know that caliper will fail with certain configurations and we can not let the whole tool to fail. run-caliper.py should handle caliper errors.
A variable must be added in the config file specifying the number of tries until we assume the execution of caliper failed.
This script should be also improved to delete the residual reports generated (possible benchmark reports created even though the execution failed) in case of failure.
If the execution of caliper completely failed after trying a number of times this script should return this information to the main script, exit(-1) should be enough.
UPDATE 24/01: As of 23/01 this task was nearly fulfilled but we have seen that the caliper command does not fail if the benchmark prints a failure. A proposed solution for this can be seen in #74

Acceptance criteria

• The code of the script runs without any errors (error handling correctly used)
• The script runs caliper and in case of caliper error handle the error and retry the number of times specified in the config file
• In case a failed caliper execution creates a failed residual report, the script must delete this report.
• The script must return to the main script the error exit code in case caliper failed after a number of retries

UPDATE 24/01

• The script detects if the benchmark run successfully or not even if the caliper command was successful.

Associated story

#22

Notes

[Subtask] Script to deploy the VM templates

---

due: 2020-01-12

Task

Detailed Description

We need a script that will deploy the VMs using the templates or script of task #33 . This script will need as parameter the number of nodes and each machine will need an account with its keystore and password from task #18

Associated story

#17

Notes

[Subtask]Create VM deployment script

---

due: 2020-01-12

Subtask

Detailed Description

We need to deploy a VM for each node, in order to make this possible we need a deployment script that will install all whats needed to run ethereum in the VM.
This script will just install the dependencies needed to run an ethereum node or a bootnode

Acceptance Criteria

• The code of the script runs without any errors (error handling correctly used).
• The script installs geth and puppeth
• Geth and puppeth can be run without issues after the execution of this script

Associated story

#17

Notes

Research bootnode metadata/information

Task

Detailed Description

We need to know if the bootnode gathers information of the nodes already connected in the network. If it gathers the information, we want to know if we can somehow retrieve it and use it in a script. This will be needed for the SUT deployment script.

Associated story

Story: #13 Ethereum SUT Deployment

Notes

The research should be documented commenting in this issue with the sources found. If possible, suggest a way of applying the solution.

Final report first part

Task

Detailed Description

Documentation must be done to deliver our project. There are some parts that can be done already:

• Relevant Literature gathered

• Introduction

• Background

Associated story

#30 Project Documentation

Notes

[Subtask] Implement OptiBench algorithm

Task

Detailed Description

We need to implement the following algorithm:
https://github.com/Cloud-Prototyping-WS-2019-20/cp_ws_1920/blob/algorithm-flowsharts/documentation/OptiBench%20Flowshart.pdf

Acceptance Criteria

• The code runs without errors.

• The code implements correctly the algorithm to obtain the optimal combination of block gas limit and block interval for the maximum TPS

Associated story

#77 #76

Notes

Ethereum SUT Deployment

Story

User story

As a user, I want the infrastructure controller automatically to build the Ethereum SUT for me.

Tasks

• #15 Research bootnode metadata/information
• #16 Research how to dynamically public miner nodes services in the same cluster
• #17 [SUT Deployment Script] Script to deploy the SUT container images
• #18 [SUT Deployment Script] Script to achieve miner dynamic additions in the SUT

Acceptance criteria

• The SUT Deployment Script should deploy a working SUT, the nodes must be participating in the network

Notes

The SUT Deployment Script is divided in 2 parts, both should be merged in one in the end (It may be added as a new task).

[Benchmark Algorithm] Implement algorithm to find the last improvement configuration value

Task

Detailed Description

We suppose that the user does not have knowledge about what is the configuration of block gas limit and block interval where he can have no more improvement.
Our tool should by benchmarking find these last improvement configuration values.
A new configuration variable should be added in the config file to indicate the minimum performance improvement difference.
E.g. If the config value is 5, the algorithm should stop after seeing less than 5% improvement between benchmarks.
We need to implement this algorithm in main.py.

Acceptance Criteria

• The code of the script runs without any errors (error handling correctly used)
• The script must be able to detect the block gas limit and block interval configuration where no more throughput improvement is found, taking into account the difference value from the config file.

Associated story

#13

Notes

[Benchmark Algorithm] Implement algorithm to find the minimum configuration value

Task

Detailed Description

We suppose that the user does not have knowledge about what is the minimum block interval and gaslimit values that he can try.
Our tool should by benchmarking and handling the crashes find these minimum values. From these minimum values it will continue benchmarking step by step until finding the last improvement configuration values.
We need to implement this algorithm in main.py.

Acceptance criteria

• The code of the script runs without any errors (error handling correctly used)
• The script must be able to detect the minimum block gas limit and block interval configuration where caliper can run without crashing or prompting errors.
• The script must be able, after detecting the minimum combination configuration, to run benchmarks with the steps defined in the configuration file until the limit configured

Associated story

#13

Notes

Workload Deployment

Story

User story

As a user, I want the infrastructure controller to automatically set up the workload to benchmark the SUT.

Tasks

• Create Caliper container image

• Create Workload deployment script

Acceptance criteria

• The Workload deployment script should deploy several Hyperledger Caliper instances (nodes) in a GKE cluster that can benchmark several SUT nodes.

Notes

Implement algorithm of #76

Task

Detailed Description

We need to implement the benchmark algorithm specified in #76.
This task is divided into 2 tasks: #80 #81

Acceptance criteria

• The algorithm executes the designed flow correctly.

• The code should handle errors correctly.

• The implementation should obtain the combination of block interval and block gas limit for the maximum TPS found in the execution

Associated story

#76 #64 #65 #14

Notes

Validate existing GCP setup based on user input.

Task

The task is to check if the GCP setup exists or not based on the user input.

Detailed Description

We add a flag in main.py: 1 or 0
(so that the user can decide to use an existing setup or clean the existing setup and create a new one.)
The default flag must be to use the existing setup.

When the user does 1: We just clean the existing setup

When the user does 0:

• we check what is currently running if he is right then we do the normal flag 0 execution

• we check what is currently running if he is wrong then we do the flag 1 execution

"If he is right" means checking on GCP mainly the number of nodes, if it matches with config.json or not.

Notes

At present, deploy-sut.sh has a flag that does this but it needs to be done as the first step when the user runs main.py.

Acceptance Criteria

• The code must run without errors.

• The script must correctly ask for the flag and accept the response correctly.

[Subtask] Implement in the main script to read config file

Task

Detailed Description

We need the main script to read from the config file the user will use to add his/her desired parameters.
Parameters that needs to read from the config file:

• Number of nodes

These parameters will also be accordingly sent to the other scripts we are creating.

Associated story

#37

Notes

Integration of current scripts to return info to main script

Subtask

Detailed Description

In this simple task we need to update the current scripts to make them return a value that will help the main script to know if their execution worked.
https://github.com/Cloud-Prototyping-WS-2019-20/cp_ws_1920/blob/master/bin/Python%202/temp.py
https://github.com/Cloud-Prototyping-WS-2019-20/cp_ws_1920/blob/master/bin/build-tag-push-docker-image.sh
The return value should be an integer value and if the execution is successful it must return 0, if there was any error it should return -1

Associated story

#37

Notes

Please change the name of the temp.py file to aggregate-html-reports.py

SUT script automation annoyances

Bug

Detailed Description

When executing the SUT it is not completely automated:

• When deleting the previous environment is asking for the user permission. We must agree if this needs a user answer or not for the sake of automation and speed.

• After creating the machines if no SSH key is found it prompts a cli to create them, needing a passphrase input 2 times.

• When accessing the just created VMs for the first time, it will ask for the passphrase added in the previous step. It will ask for it for each machine we are trying to access.

Associated story

Notes

Change optibench algorithm to avoid repeating find_minimum_gaslimit()

Task

Detailed Description

Right now the algorithm is trying to find the minimum gas limit for every interval but when testing the algorithm we have seen that the minimum gas limit for every interval is pretty identical and this calculation takes a lot of time.
The idea is to change optibench algorithm under find_optimal_parameters function to use the first min gas limit value for any interval.
UPDATE 04/02:
We need to improve this algorithm, research fast if the algorithm can be improved estimating the gas of the workload contract. If not take the previous minimum gas limit as a reference to calculate the new one.

Acceptance Criteria

• The code must run without errors

• The algorithm should calculate the minimum gas limit in the first interval and then be used in the following intervals, avoiding the calculation that is being done now every interval iteration.

Associated story

#88

Notes

Create Caliper container image

Task

Detailed Description

We need to create a caliper container image that will be used to deploy it in a cluster. We need to take into account that it will need configuration template files that will be changed while deploying.
Parameters that will be received by this image:

• IP and port of a node

• Account address

• Account password

Associated story

Story: #22 Workload Deployment

Notes

[Subtask] Validate/Check if we can hit several nodes with caliper

Task

Detailed Description

We want caliper to hit every node. We need to make sure if we can execute several caliper instances in the same machine or if we can not have several caliper instances in the same machine

Associated story

Notes

Pre-requisites script

Task

Detailed Description

We need to ease the installation of the tool to the user. We are already installing libraries and dependencies and we need to take care of them to facilitate the installation of the user.
We need a script that:

• Checks that Google SDK is installed, if not print a link to show where to download it / how to install it. If it is, configure the project written by the user in the config file.
• Check if geth and puppeth(docker also needed to be installed) are installed, if not print a link to show how to install it.
• Check if jq (used in the SUT script) is installed, if not print a link to show how to install it.
• Check if npm is installed. If not print a link how to install it.
• Check if caliper is installed under /bin. If not, it will be installed with npm.
• Run caliper bind to configure ethereum.
• Execute the template creation script #61
• If possible in this script, check that the python libraries needed are installed. If possible find an easy way to install these libraries. If it is not possible it should be enough with checking if these are installed or not.

Acceptance criteria

• The code of the script runs without any errors (error handling correctly used)

• The code should print clear steps of how to install or at least show a link with clear documentation of how to install what is needed.

• If everything is installed, show a message that dependencies are fulfilled.

• If after running the script everything is fulfilled, main.py must be able to run correctly without dependencies errors

Associated story

#30 #72

Notes

the script is in #72

[SUT Deployment Script] Script to achieve miner dynamic additions in the SUT

Task

Detailed Description

We need a script that deploys the Ethereum SUT.
This part of the script should receive a notification of GCP whenever a new node is set up and running. After receiving such notification it will communicate with the nodes already sealing to accept this new node as a new sealer.
Update 17-12-19:
After discussion, we decided to add the sealers with the genesis file. The script should get the number of nodes, gas limit and block interval as argument and create the same number of accounts as nodes locally with geth. Next it will create the genesis file taking block interval and gas limit as well as the accounts that should be added as sealers. The final output for this part of the script would be the genesis.json file and the account keystores that later on will be built in the node containers.

Associated story

Story: #13 Ethereum SUT Deployment

Notes

This task is open for discussion depending on the results of #15 and #16

Implement main script logic to execute the SUT and Workload script

Task

Detailed Description

With the SUT script #47 and the workload script #46 the main script can now execute the logic needed to benchmark.
The main script should follow this loop logic:

1. Build the SUT using the SUT script. Take into account this script needs parameters.

2. Execute the workload using the workload script. Take into account this script also needs parameters.

3. Wait for the results of caliper benchmarking.

The loop should change the values of block interval and gas limit for each iteration.

Associated story

#13 #22

Notes

Find useful literature that research about block gas limit and block interval impact.

Task

Detailed Description

We need to know if there is already literature that researched about the block gas limit and block interval impact over throughput. Specifically, if there is a formula or model to apply in our solution.
Update 17/12/19: As discussed in group the information found already confirms the influence of block gas limit and block time over throughput. However, the paper does not cover PoA with block interval. Further research is needed to find more related work.

Associated story

Story: #14 Block gas limit and block interval optimal values

Notes

The research should be documented here in a comment with the sources found. If possible, suggest a way of applying the solution.

Test Issue

---

due: 2020-01-14

Task

Detailed Description

Associated story

Notes

[SUT Deployment Script] Script to deploy the SUT container images

Task

Detailed Description

We need a script that deploys the Ethereum SUT.
This part of the script should take the container images published in Google Container Registry and use them to deploy the SUT.
The nodes should be accessible through an IP and port.
This script should receive as parameter the number of nodes, the block gas limit and block interval.

Associated story

Story: #13 Ethereum SUT Deployment

Notes

First step of this task should be to create the YAML Kubernetes template for the deployment of nodes

Help sources

https://imti.co/ethereum-kubernetes/
https://cloud.google.com/container-registry/docs/using-with-google-cloud-platform#google-kubernetes-engine