chaosreload/stress-test-loader

seconddinner stress-test-loader

Second Dinner stress-test-loader

This is the Second Dinner stress-test-loader. It is a small golang application for executing stress tests, plus some infra things (packer, terraform, etc) to deploy it to the cloud. We have open-sourced this code to contribute to development community.

Currently, this setup targets AWS, but it can be ported other clouds if needed.

Directory structure:

• stress-test-loader (golang service that can load any stress-test-client, plus packer templates for creating AMIs)

• infra-terraform (terraform config for deploying AMIs to create stress-test-loader ec2 instances)

Requirements

1. golang.
2. hashicorp packer.
3. configure your AWS environment keys.
4. AWS Cli
5. hashicorp terraform.

stress-test-loader

To take advantage of AWS arm64 offering, we are building the golang executable and packer AMI image all in arm64 format.

The following example demonstrates building the stress-test-loader ami.

Generate AMI

1. cd stress-test-loader
2. source cicd/ami/build-stress-test-loader.sh
3. If everything worked according to plan, you will see message like AMIs were created: ami-XXXXXXXXXXXXXXX

Infra-terraform

Once you have created an AWS AMI for stress-test, you can use Infra-terraform to create EC2 instance and create as many EC2 instances as your AWS account allows.

terraform stress infrastructure

1. make a copy of infra-terraform/instance/single-region in infra-terraform/instance/, we use infra-terraform/instance/single-region-test as an example.
2. replace backend "s3" in infra-terraform/instance/single-region-test/main.tf with your own backend.
3. need following variables: public_key, your aws account id as owner_id, your machines public ip as stress_test_loader_allowed_cidr
4. update variables infra-terraform/instance/single-region-test/variable.tf.
5. cd infra-terraform/instance/single-region-test
6. terraform init
7. terraform apply
8. If everything worked according to plan, you will see message like Apply complete! Resources: XX added, XX changed, XX destroyed

Running the stress test using stress-test-loader

Create stresstest client json

• build your stress test client as an arm64 executable, this can be a directory with libraries and one entry executable. The executable can take any number of environment variable as configuration. We are going to use stress-test-client as an example
• tar cvzf stress-test-client
• copy the tgz file to an S3 bucket aws s3 cp stress-test-client.tgz s3://stress-test-client-s3/stress-test-client.tgz
• build a stress-test-loader config json. For example stresstest.json

{
    "s3": "stress-test-client-s3",
    "s3key": "stress-test-client.tgz",
    "loadtestExec": "StressTest.Cli",
    "envVariableList": [
        {
            "EnvName": "TargetEnvironment",
            "EnvValue": "EnvValue"
        },
        {
            "EnvName": "SomeOtherEnv",
            "EnvValue": "us-west-2"
        }
    ]
}

• export all of the ec2 instances' public IP addresses aws ec2 describe-instances --region us-west-2 --query 'Reservations[*].Instances[*].{"public_ip":PublicIpAddress}' --filters Name=instance-state-name,Values=running --output json > /tmp/IP.json
• run stress test cd stress-test-loader/client; go run main.go stresstest.json /tmp/IP.json
• if you gave an ssh public key, you can ssh into the ec2 instance and check its systemd service log journalctl -f -u stress*