unfor19/terraform-multienv

A template for maintaining a multiple environments infrastructure with Terraform. This template includes a CI/CD process, that applies the infrastructure in an AWS account.

terraform-multienv

A template for maintaining a multiple environments infrastructure with Terraform. This template includes a CI/CD process, that applies the infrastructure in an AWS account.

environment	drone.io	GitHub Actions	Circle Ci	Travis CI
dev
stg
prd

Assumptions

• Branches names are aligned with environments names, for example dev, stg and prd

• The CI/CD tool supports the variable ${BRANCH_NAME}, for example ${DRONE_BRANCH}

• The directory ./live contains infrastructure-as-code files - *.tf, *.tpl, *.json

• Multiple Environments

  • All environments are maintained in the same git repository
  • Hosting environments in different AWS account is supported (and recommended)

• Variables

  • ${app_name} = tfmultienv
  • ${environment} = dev or stg or prd

Getting Started

1. We're going to create the following resources per environment

   • AWS VPC, Subnets, Routes and Routing Tables, Internet Gateway
   • S3 bucket (website) and an S3 object (index.html)
   • Terraform remote backend - S3 bucket and DynamoDB table

2. Create a new GitHub repository by clicking - Use this template and don't tick Include all branches

3. AWS Console > Create IAM Users for the CI/CD service per environment

   • Name: ${app_name}-${environment}-cicd
   • Permissions: Allow Programmatic Access and attach the IAM policy AdministratorAccess (See Recommendations)
   • Create AWS Access Keys and save them in a safe place, we'll use them in the next step

4. GitHub > Create the following repository secrets for basic application details

   • APP_NAME - Application name, such as tfmultienv
   • AWS_REGION - Region to deploy the application, such as eu-west-1 (Ireland)

5. GitHub > Create the following repository secrets for authenticating with AWS, according to the access keys that were created in previous steps

   • AWS_ACCESS_KEY_ID_DEV
   • AWS_SECRET_ACCESS_KEY_DEV
     IMPORTANT: The names of the secrets are not arbitrary, make sure you set them as shown in the example below

6. Deploying the infrastructure - Commit and push changes to your repository

   git checkout dev
   git add .
   git commit -m "deploy dev"
   git push --set-upstream origin dev
   

7. Results

   • Newly created resources in AWS Console - VPC, S3 and DynamoDB Table
   • CI/CD logs in the Actions tab (this repo's logs)
   • The URL of the deployed static S3 website is available in the terraform-apply logs, for example:
     1. s3_bucket_url = terraform-20200912173059419600000001.s3-website-***.amazonaws.com
     2. Replace *** with the AWS_REGION, for example eu-west-1
        terraform-20200912175003424600000001.s3-website-eu-west-1.amazonaws.com

8. Create stg branch

   git checkout dev
   git checkout -b stg
   git push --set-upstream origin stg
   

9. GitHub > Promote dev environment to stg

   • Create a PR from dev to stg
   • The plan to stg is added as a comment by the terraform-plan pipeline
   • Merge the changes to stg, and check the terraform-apply pipeline in the Actions tab

10. That's it, you've just deployed two identical environments! Go ahead and do the same with prd

11. How to proceed from here

    1. Make changes in dev - commit and push
    2. Promote dev to stg - create a PR
    3. Promote stg to prd - create a PR
    4. Revert changes in dev - reverting a commit
    5. Revert changes in stg and prd - reverting a PR

Recommendations

Generic

• Naming Convention should be consistent across your application and infrastructure. Avoid using master for production. A recommended set of names: dev, tst (qa), stg and prd. Using shorter names is preferred, since some AWS resources' names have a character limit
• Resources Names should contain the environment name, for example tfmultienv-natgateway-prd
• Terraform remote backend costs are negligible (less than 5$ per month)
• Using Multiple AWS Accounts for hosting different environments is recommended.
  The way I implement it - dev and stg in the same account and prd in a different account
• Create a test environment to test new resources or breaking changes, such as migrating from MySQL to Postgres. The main goal is to avoid breaking the dev environment, which means blocking the development team.

Terraform

• backend.tf.tpl - Terraform Remote Backend settings per environment. The script prepare-files-folders.sh replaces APP_NAME with TF_VARS_app_name and ENVIRONMENT with BRANCH_NAME
• Remote Backend is deployed with a CloudFormation template to avoid the chicken and the egg situation
• Locked Terraform tfstate occurs when a CI/CD process is running per environment. Stopping and restarting, or running multiple deployments to the same environment will result in an error. This is the expected behavior, we don't want multiple entities (CI/CD or Users) to deploy to the same environment at the same time
• Unlock Terraform tfstate by deleting the items from the state-lock DynamoDB table, for example
  • Table Name: ${app_name}-state-lock-${environment}
  • Item Name: ${app_name}-state-${environment}/terraform.tfstate*

Security

• AdministratorAccess Permission for CI/CD should be used only in early dev stages. After running a few successful deployments, make sure you restrict the permissions per environment and follow the least-previleged best practice. Use CloudTrail to figure out which IAM policies the CI/CD user needs, a great tool for that - trailscraper
• IAM Roles for self-hosted CI/CD runners (nodes) are preferred over AWS key/secret

Git

• Default Branch is dev since this is the branch that is mostly used
• Branches Names per environment makes the whole CI/CD process simpler
• Feature Branch per environment complicates the whole process, since creating an environment per feature-branch means creating a Terraform Backend per feature-branch.

Repository Structure

• Modules should be stored in a different repository
• Infrastructure Repository should be separated from the Frontend and Backend Respositories. There's no need to re-deploy the infrastructure each time the application changes (loosely coupled)

References

• To get started with Terraform, watch this webinar - Getting started with Terraform in AWS
• Terraform Dynamic Subnets
• Terraform Best Practices - ozbillwang/terraform-best-practices

Authors

Created and maintained by Meir Gabay

License

This project is licensed under the MIT License - see the LICENSE file for details

terraform-multienv's Issues

Add PR approval mechanism

1. User pushes code to dev - terraform apply on dev (already exists)
2. User creates a PR from dev to stg - terraform apply on stg`
3. GitHub Bot Comments a readable format of the plan in the PR
4. User approves PR merge - terraform apply on stg (already exists)

Align all CI/CD services with GitHub Actions

Currently, only the GitHub Actions pipeline is working according to the new PR approval process.

The following CI/CD services need to be updated

• drone.io
• CircleCi
• TravisCI

Refactor Bash scripts

Current scripts are not organized with functions, making them less readable, and less modular.

Migrate the main example from drone.io to GitHub Actions

Current flow for newcomers

1. AWS - create IAM users
2. GitHub - use template, find and replace values
3. drone.io - create a user, activate repository, and create secrets

A better flow

1. AWS - create IAM users
2. GitHub - use template, find and replace values, and create secrets

The service drone.io is amazing. This migration is only for making it easier for newcomers to use this template.

Add notification step when deploying to stg and prd

1. On push (merge) to stg and prd, execute terraform apply (already exists)
2. Add a notification step at the end of the pipeline - try to make it abstract, so it can be used with multiple services. For starters - Slack channel

Migrate hardcoded values to secrets - app_name and region

Find and replace is not so nice, especially when you can't enforce it. It's better to convert app_name and region to secrets. This way, if the secrets are not provided, the pipeline exits.

Add more modules and resources

The current example only presents a VPC module. Proving that this template is useful will be easier when it contains more resources and modules, and a larger variables.tf file

Add issues templates and labels

Add CONTRIBUTING.md

Detailed steps of how to contribute to this project