sravanmummadi/awesome-scalability

clone of https://github.com/binhnguyennus/awesome-scalability

aws-kafka

TerraForm Scripts for AWS and Kafka

To use this:

You must set these environmental variables to use this $ export AWS_ACCESS_KEY_ID="anaccesskey" $ export AWS_SECRET_ACCESS_KEY="asecretkey" $ export AWS_DEFAULT_REGION="us-east-1"

You will be asked for an existing key in each region: set or know before you run apply

Variables are driven from within the production environment: aws_kafka/produciton/services/kafka-cluster/vars.tf or from commandline

key_name = ssh key that has already been uploaded to the AWS region that this is targeting cluster_name - unique name for the cluster - default = "kafka-01" broker_instance_type - broker instance type - default = "t2.micro" zookeeper_instance_type - zookeeper instance type- default = "t2.micro" number_of_zookeepers - number of zookeeper nodes to build default = 3 number_of_brokers - number of brokers to build default = 6

Outcome:

This will build a multi node zookeeper and broker configuration with appropirate security groups in your default VPC

TODO:

• Add ansible for Kafka build - started, need to expand for full build
• Custom AMI to allow ansible to work on remote nodes without intervention - done
• dynamic inventory in ansible based on tags generated by terraform - done
• put in ELB in front of Kafka nodes
• put in autoscaling for broker nodes
• fix the azs variables to expand to dynamic AZ list