eosnetworkfoundation/chicken-dance

Chicken Dance distributed replay of transactions

chicken-dance's Issues

Basic Authentication via username password

Depends on Issue #20 Nginx setup. Will be configured in Nginx.

Create a basic username/password authentication

• admin role
• readonly role

Cron for replay node jobs

Add a cron job for replay nodes to get new jobs. Must include lock file to prevent more than one job running at the same time on host.

Share Orchestrator Private IP with Replay Nodes

When the orchestrator starts up the relay hosts need to connect back to the orchestrator. The current security group settings only allows replay nodes to connect with the private ip of the orchestrator node.

There are several ways to address this problem

DNS

• you can use DNS to map a name to a private IP address, the relay hosts would simply connect using this DNS name
• you can use DNS to map a name to a public IP address, and change the security to allow access from public interfaces of the relay nodes

AWS CLI

• you can script the orchestrator node to spin up the relay nodes and then share the ip address with the relay nodes.
  • could share by passing an argument into a script
  • could share by placing the private IP into a file on the relay node

[REPLAY-18] UI: Start Run

UI Work only

Create Report Across all runs after full replay is complete

So we have the integrity hashes both expected and actual, along with start, end block numbers. Not sure what report folks want to see when the full replay has completed.

Better Error Message for Config Update

When the service tries to update a configuration that doesn't exist all you see on the log output is '404'. Need to improve this error message to say something like config at block height X does not exist

Create two AWS hosts for master syncing

Transferred from AntelopeIO/leap#1427 original author bhazzard

Support different EOS Chains via configuration

Add a new meta-data configuration to specify the chain. Initially these would be

• test
• mainnet
• jungle

Clean up Replay Script Output

Clean up output from replay script. So if you watched it from the command line it would make more sense. This includes

• directing verbose output to log away from console
• echoing out status as progress is made

This is a low priority

Replay Nodes Work with Orchestration Service

• startup http/get job with config (jitter and backup to prevent congestion)
• copy over snapshot
• startup with snapshot
• update http/post job

[REPLAY-2] Orchestration Service supports option to prefer configured integrity hashes

By default, expected integrity hashes are overwritten. Provide --use-configured-hash option when starting orchestration service. When enabled

• If the configuration has an expected integrity hash this will be used to validate replay
• The replay script will understand this option, and skip the steps of posting back start block integrity hash
• The replay script will instead directly load the snapshot and sync until end block
• If the configuration does not have a value for expected integrity hash the normal process will continues as before
• The replay script will loading snapshot, terminate nodeos, and starting read only nodes, and posting back start block integrity hash
• Following that the nodes will continue to sync to end block, and continue normal operations

Another option to --protect-configured-hash is similar. It will make sure the expected hash with a value can not be overwritten with None. It will allow new hashes values to be set over the old value.

[REPLAY-9] Generate Multi-Version Manifest

Takes arguments for start_block_id, end_block_id, snapshot_path, snapshot_type, and expected_integrity_hash, and generate configuration for leap version from 3.1, 3.2, 4.0, 5.0.

Alternatively takes arguments for a config file, and a slice-id and generates the same by parsing the file for the specified index.

Manifest should generate job to start at genensis

Summary report should re-test HASH-MISMATCH jobs

The expected integrity hash depends on another replay job completing. Sometimes that job is slow and the expected integrity hash does not arrive arrive in time. The empty hash causes the job to be listed as a failure. These HASH-MISMATCH jobs should be retested, and labeled as COMPLETE if there hashes do match

Gather snapshots into 630 roughly equal chunks

Transferred from AntelopeIO/leap#1421 original author bhazzard

[REPLAY-10] Oauth Login to Orchestration Service Via Github

Requires a single stable orchestration instance with a fixed IP address. The idea is to enable authentication via github oauth and check for membership in a specific github group. This would provide a way to log into the orchestration without a new username/password.

[REPLAY-5] HTTPS access DNS A-records and certificate

Support secure HTTPs accesses. Caveat this will only support a single fixed orchestration node, used to test production runs.

Customized runs can not be done over HTTPs. Customized runs will require building a configuration file, spinning up a new orchestration service. A new orchestration service will not have a pre-determined HTTP address and will not have a host-domain in DNS.

Once DNS is setup, run lets encrypt to generate certificates for secure connection.

[REPLAY-19] UI: Load Run

UI Work only

[REPLAY-7] log rotate orchestration log

setup log rotation for /home/ubuntu/orch-complete-timings.log on orchestration node.

Gather blockslog from Genesis to Current

Transferred from AntelopeIO/leap#1441 original author BenjaminGormanPMP

EC2 Run Instance does not work when AIM role specified

Steps to Reproduce

1. enter chicken-dance aws organization
2. start an instance using the template LowEndOrchestrator
3. ssh into the instance and run the following to setup the aws cli

# setup aws cli
sudo apt update
apt install unzip
sudo apt install unzip
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" && unzip awscliv2.zip
sudo ./aws/install

4. Using the dry-run option try to start a new node. Only works when --iam-instance-profile is removed.

aws ec2 run-instances --image-id ami-053b0d53c279acc90 \
--instance-type t2.micro \
--security-group-ids sg-04f895bd6442b69b5 \
--key-name aws-chicken-start \
--subnet-id subnet-0b9517f11e9684b3b \
--iam-instance-profile Arn=arn:aws:iam::087045697350:instance-profile/ReplayOrchestration \
--user-data file:///home/ubuntu/replay-test/scripts/instance-bootstrap.sh \
--dry-run

Expand IAM for Zero-Trust S3-EC2 Integration

Following leap issue 1420 and preceding replay-test issue 11, this ticket is to expand the IAM policy in the chickens-prod AWS account to enable @ericpassmore and team to grant zero-trust access of S3 bucket(s) to their EC2 instances.

Fixed IP Address for main Orchestration Service

Create a stable fixed, public IP address for the main orchestration node. Any other orchestration nodes spun up will not be covered and their public IP address should be considered unstable.

Post integrity hash from snapshot back to orchestration service

Create post for integrity hash after snapshot loads. Posts updates configuration, and persists configuration file

[REPLAY-16] UI: Manage Run Page

Page to manage jobs.

• If job running option to stop

• If no job running and to Start Job, enter number of hosts, default 80% num slices

• If no job running option to create job

• input job name

• drop down of config available

• option to overwrite leap version

• option to copy and past in new config

Create mainnet manifest for snapshots and block positions

Transferred from AntelopeIO/leap#1422 original author bhazzard

Create orchestration service

Due to a bug need a http service to inform replay-nodes of run options, and gather status information from nodes.

Tasks

Beta Give feedback

1. Write Python HTTP web service
2. Get Call job, no args, returns get json below
3. Put Call status, url has Job Number, pushed json

Get Call

{
job_number: num,
snapshot_loc: string,
blocklog_loc: string,
last_block: number
}

Put Call
{ status:[ongoing|stopped|error|finished], block_number: last_block_num_processed }

Orchestration Proxy Though Real HTTP

Setup Nginx or Apache to proxy the orchestration webservice. For better HTTP support, keepalive, and authentication/access congtrol

[REPLAY-12] UI: Create Home Page CSS/HTML

Create basic HTML CSS for homepage. This includes .

### Tasks
- [ ] foreground, background, accent, shadow colors
- [ ] fonts 
- [ ] media sizes for phone 360x800, desktop 1280x720, HD 1920x1080
   - [ ] padding and margins 
   - [ ] text sizes header, body, strong, 
   - [ ] nav cards 
   - [ ] lists

[REPLAY-6] API authentication to orchestration service

There will be a single fixed orchestration service accessible by HTTP and a domain name. Depends on Issue #35

This issue covers access control for an API to reach this single host, start a pre-configured run, and collect status on that run.

Test with single replay node

Transferred from AntelopeIO/leap#1429 original author bhazzard

Enable multiple runs on a replay node

Enable a replay to pick up a job, process it, clean out state, and pick up a new job and process it. This is needed because some jobs will take longer than others, and we want to leverage all our compute power if there are any jobs WAITING_4_WORKER

[REPLAY-8] Scale Up Option with Zone Agnostic Placement

Want to scale up the amount of compute. Our current launch template for replay has a subnet with a specific placement zone. We want to explore ways get more capacity by spinning up nodes in many placement zones

[REPLAY-20] Manual Update Blocklogs on S3

Tasks

Beta Give feedback

1. spin up an empty replay host, disable cron
2. get list of blocks logs push to file
3. iterate through list create new blockslog with 2 million stride
4. test and validate new blockslog
5. compress and upload

Then automate a batch of 5 , then automate larger run.

[REPLAY-3] More descriptive Log messages for Job updates

Currently show HTTP 200 , Success. Need a more descriptive message.

Gather blockslog from Genesis to Current

Transferred from AntelopeIO/leap#1441 original author BenjaminGormanPMP

Write validate scripts

Transferred from AntelopeIO/leap#1430 original author bhazzard

[REPLAY-11] UI: Create Top Nav Component

New Top Nav HTML and CSS. Static and served by front-end Nginx, not by webservice.

### Tasks
- [ ] first pass icons and html in repo
- [ ] orchestration bootstrap copy in HTML, CSS, Icons 
- [ ] visual test inspection 

Create two AWS hosts for replay

Transferred from AntelopeIO/leap#1423 original author bhazzard

All installation of software from deb packages. Installation of config from this repo

Tasks

Beta Give feedback

1. AWS template for hosts
2. Init Shell script to create linux user, setup dirs, install nodeos
3. Create default config file
4. Create startup script for nodeos
5. Create shutdown script for nodeos
6. make genesis file avalible

Correct Set Start Time for Relay Processing

When job status is set to "STARTING" update the job start time. Currently start time is set to initialization of orchestration webservice.

[REPLAY-1] Run Replay From Genesis to Current

Must do full run in less than 1 day. Ideally complete in 16 hours or less. Transferred from https://github.com/AntelopeIO/leap/issues/1431 original author bhazzard

Generate Statistics on Job Times

Example Report below

JOB TIMING ALL TIMES IN MINUTES
-------------------------------
Number of Jobs: 222
Average: 99.73
Standard Deviation: 160.04
Median: 61.77
75th Percentile: 90.98
90th Percentile: 171.06
Longest Running Job 1333.6 mins

LONG RUNNING JOBS TOP 90%
-------------------------
Job 140648764956560 running time 171.1
Job 140648764956848 running time 177.4
Job 140648764956944 running time 212.58
Job 140648764957040 running time 230.88
Job 140648764957136 running time 240.62
Job 140648764957232 running time 248.12
Job 140648764957328 running time 247.9
Job 140648764957424 running time 285.37
Job 140648764957616 running time 273.65
Job 140648764957520 running time 302.13
Job 140648764957760 running time 252.17
Job 140648764957712 running time 281.08
Job 140648764957856 running time 267.17
Job 140648764957808 running time 522.9
Job 140648764957952 running time 493.6
Job 140648764958048 running time 477.83
Job 140648764958240 running time 474.8
Job 140648764957904 running time 742.62
Job 140648764958144 running time 804.1
Job 140648764958000 running time 1006.08
Job 140648764958432 running time 669.4
Job 140648764958336 running time 852.53
Job 140648764958096 running time 1333.6

[REPLAY-21] Pull Down Correct Block Logs with 2 million stride

search for matching blocks log

• if exact match start/end take it
• if inside range take it
• if spanning range take both and combine on host

Create script to update last block processes during replay

A little script to run in the background and update the job with the head_block_num every few minutes. Will allow you to see progress as blocks update.

[REPLAY-15] UI: Summary Progress

Create Summary Progress Page

### Tasks
- [ ] Construct Layout
- [ ] Color Change Progress Bar (attribution)
- [ ] Top Bar for blocks process and progress
- [ ] Second Bar icon key for jobs , succeed, failed , total
- [ ] Update progress stats python to produce HTML
- [ ] Third Content Card with stats (avg, median, stddev of times)
- [ ] Forth simple list of list failed jobs, larger font size

EX: progress bar https://codepen.io/alvarotrigo/pen/vYeNpjj

[REPLAY-13] UI:Single Item Detail Page

Create a Single Item Detail Page

• Add classes to html

• add Top Grey Banner Style

• Visual Inspection

Test Run a replay on Leap v4.04 (run on 10 slices to speed up replay)

Transferred from AntelopeIO/leap#1426 original author bhazzard

[REPLAY-14] UI: Listing Items Details

Create items listing page.

### Tasks
- [ ] Change HTML structure to allow float
- [ ] Visual inspect multiple columns depending on screen width

[REPLAY-4] Compress Cloud Stored Blocks.log

When archiving blocks.log compress it. When retrieving blocks.log uncompress

[REPLAY-17] UI: Stop Run

UI Work Only