swoodford/aws

A collection of bash shell scripts for automating various tasks with Amazon Web Services using the AWS CLI and jq.

A collection of bash shell scripts for automating various tasks with Amazon Web Services using the AWS CLI and jq.

https://github.com/swoodford/aws

Table of contents

• Why
• Getting Started
• What's Included
• Bugs and Feature Requests
• Creator
• Copyright and License

Why

Why does this project exist?

This repository is intended to make some of the more difficult DevOps tasks commonly required to maintain complex hosting infrastructure in AWS simple, quick and easy. This is my attempt to automate and expedite difficult, repetitive, tedious and time consuming processes into a simple shell script that gets the job done as cleanly as possible. These scripts were developed out of frustration in clicking around on the same things over and over again in the web console every day, week, month when they could easly be done in seconds in a script that uses the AWS CLI. I've tried to keep everything applicable to as many use cases across regions and across as many different AWS accounts as possible. I run many of these scripts myself, mostly on a Mac or in Linux and do periodic usability and bug checking, making updates for any changes to the CLI. I hope this collection of tools helps you too, and if you use these please hit the Star/Fork button and if you have any suggestions please open an Issue or PR!

Getting Started

What is the AWS Command Line Interface?

The AWS CLI is an open source tool built on top of the AWS SDK for Python (Boto) that provides commands for interacting with AWS services.

Installing the AWS Command Line Interface

Requirements:

• Python 2 version 2.6.5+ or Python 3 version 3.3+
• macOS, Linux, or Unix

If you already have pip and a supported version of Python, you can install the AWS CLI with the following command:

$ pip install awscli --upgrade --user

Configuring the AWS CLI

For general use, the aws configure command is the fastest way to set up your AWS CLI installation.

$ aws configure

The AWS CLI will prompt you for four pieces of information. AWS Access Key ID and AWS Secret Access Key are your account credentials.

Named Profiles

The AWS CLI supports named profiles stored in the config and credentials files. You can configure additional profiles by using aws configure with the --profile option or by adding entries to the config and credentials files.

$ aws configure --profile example

What is jq?

jq is a lightweight and flexible command-line JSON processor.

Installing jq

OS X: Use Homebrew to install jq:

$ brew install jq

Linux: jq is in the official Amazon Linux AMI, Debian and Ubuntu repositories.

Amazon Linux AMI, RHEL, CentOS:

$ sudo yum install jq

Debian/Ubuntu:

$ sudo apt-get install jq

Tools included in this repo:

CloudFront

• cloudfront-inprogress-status.sh Checks CloudFront Distributions with In-Progress status to complete and become Deployed
• cloudfront-invalidation-status.sh Checks CloudFront Distributions for cache invalidation status to complete

CloudWatch

• cloudwatch-create-alarms.sh Create CloudWatch alarms for EC2, RDS, Load Balancer environments
• cloudwatch-create-alarms-statuscheckfailed.sh Create CloudWatch StatusCheckFailed Alarms with Recovery Action for all running EC2 Instances in all regions available
• cloudwatch-create-alarms-unhealthyhost.sh Create CloudWatch UnhealthyHost Alarms for all ALB and ELB Elastic Load Balancers in all regions available
• cloudwatch-logs-cleanup.sh Delete all CloudWatch Log Groups with a Last Event that is older than the Retention Policy
• cloudwatch-logs-delete-groups.sh Quickly delete all CloudWatch Log Groups with a specified prefix in all regions available
• cloudwatch-logs-search.sh Search CloudWatch Logs for any string across all regions and log groups
• cloudwatch-logs-retention-policy.sh Set CloudWatch Logs Retention Policy for all log groups in all regions available

EC2

• ec2-ami-encrypted-ebs-boot-volume.sh Create an AMI with an encrypted EBS boot volume from the latest Amazon Linux AMI
• ec2-associate-elastic-ip.sh Reassign a previously allocated Elastic IP to the instance which runs this script
• ec2-classic-import-network-acl.sh Import CIDR IP list to AWS EC2 Classic ACL rules and deny access
• ec2-ebs-create-snapshots.sh Create a snapshot of each EC2 EBS volume that is tagged with the backup flag
• ec2-ebs-delete-snapshots.sh Deletes snapshots for each EC2 EBS volume that is tagged with the backup flag and matches the specified date
• ec2-elb-export-template.sh Export an ELB to a JSON template file for version control, duplication or recreation
• ec2-elb-upload-ssl-cert.sh Upload an SSL Certificate to AWS for use in setting up an ELB

Elastic Beanstalk

• elastic-beanstalk-set-hostname-within-instance.sh Set the hostname on Elastic Beanstalk servers from within the instance with their EB environment name and public IP address
• elastic-beanstalk-update-hostnames.sh Updates the hostname on Elastic Beanstalk servers with their environment name and IP address

IAM

• iam-create-s3-users.sh Create the S3 IAM user, generate IAM keys, add to IAM group, generate user policy

Route53

• route53-export-zones.sh Uses cli53 to export the zone file for each Hosted Zone domain in Route 53

S3

• s3-buckets-local-backup.sh Backup all contents of all S3 buckets in AWS account locally
• s3-buckets-security-audit.sh Export S3 bucket ACL, CORS, Policy and Website as JSON for auditing security of all buckets
• s3-buckets-total-file-size.sh Count total size of all data stored in all S3 buckets using s3api
• s3-fix-content-type-metadata.sh Safely fix invalid content-type metadata on AWS S3 bucket website assets for some common filetypes
• s3-open-bucket-policy.sh Set an S3 bucket policy to allow GetObject requests from any IP address (publicly accessible website)
• s3-remove-glacier-objects.sh Delete all Glacier storage type objects in a single S3 bucket
• s3-restrict-bucket-policy.sh Set an S3 bucket policy to only allow GetObject requests from an IP whitelist file named iplist
• s3-set-cache-control-max-age.sh Set Cache-Control max-age value on AWS S3 bucket website assets for all filetypes
• s3-setup-buckets.sh Create S3 buckets, set CORS config and tag bucket with client name

VPC

• vpc-eni-monitor.sh Generate an HTML page to monitor the number of AWS VPC Elastic Network Interfaces currently in use and upload it to an S3 bucket website
• vpc-sg-import-rules.sh Create an AWS VPC Security Group with rules to allow access to each IP at the port specified
• vpc-sg-import-rules-cloudflare.sh Create VPC Security Group with Cloudflare IP ranges
• vpc-sg-import-rules-cloudfront.sh Create VPC Security Group with CloudFront IP ranges
• vpc-sg-import-rules-pingdom.sh Create or Update VPC Security Groups with Pingdom probe server IP ranges
• vpc-sg-merge-groups.sh Merge two existing VPC Security Groups together into one single group
• vpc-sg-rename-group.sh Rename an existing VPC Security Group by creating an identical new group

WAF

• waf-export-ip-sets.sh Export each AWS WAF IP set match condition to a JSON file for backup
• waf-import-ip-set-facebook.sh Import list of current Facebook crawl server IPs into AWS WAF IP Set - work in progress, currently not possible to execute
• waf-web-acl-pingdom.sh Manage WAF Web ACL to allow current Pingdom probe server IPs by creating or updating AWS WAF IP Addresses Set, Rules and Web ACLs

Other Tools

• convert-iplist-cidr-json-array.sh Converts an IPv4 iplist to CIDR block notation and JSON array format, sorting and de-duplicating IPs
• install-awscli.sh Install and configure AWS CLI
• install-s3cmd.sh Install and setup s3cmd from the GitHub Repo
• terraform-redact-iam-secrets.sh Replaces AWS IAM Secret Keys and IAM SES SMTP Passwords with "REDACTED" in Terraform state files

Bugs and feature requests

Have a bug or a feature request? The issue tracker is the preferred channel for bug reports, feature requests and submitting pull requests. If your problem or idea is not addressed yet, please open a new issue.

Creator

Shawn Woodford

• https://shawnwoodford.com
• https://github.com/swoodford

Copyright and License

Code and Documentation Copyright 2012-2018 Shawn Woodford. Code released under the Apache License 2.0.

aws's Issues

Enable flag to bypass logs with retention already set

Enable 'overwrite' flag, basically do you want to overwrite an existing retention period.
This would be helpful in case some logs already have a policy of say 30 days, but for buckets with no policy defined we want a max of 180 days retention.

aws/cloudwatch-logs-retention-policy.sh

Line 10 in fd6fc2b

Add any other context or screenshots about the feature request here.

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Schedule

Day1

Elastic Beanstalk EnvironmentHealth Alarm

Your tools look awesome! 👏

Is there a way to automate the creation of an EnvironmentHealth alarm for a given Elastic Beanstalk application? I need to create the exact same alarm for a host of applications, each of which has multiple environments.

e.g.
I have foo-app with foo-app-production and foo-app-staging
I have bar-app with bar-app-production and bar-app-staging

All 4 of those need to message a SNS topic on both Alarm and OK.

Test

Don't Repeat Yourself

Is your feature request related to a problem? Please describe.
Repeating yourself

Describe the solution you'd like
Pull the configuration into a single file

Describe alternatives you've considered
N/A

Additional context
N/A

You have this atop all of your scripts and if this ever changes, for whatever reason (which it probably won't), you'll be copy/pasting it all over again throughout. Also if someone wants to add some initialization steps beyond what is specified here you may benefit them as they could simply add to this included file. So instead of this:

if ! grep -q aws_access_key_id ~/.aws/config; then
  if ! grep -q aws_access_key_id ~/.aws/credentials; then
    echo "AWS config not found. Running \"aws configure\"."
    exit 1
  fi
fi

Move it to _pre_command.sh and improve it:

#!/usr/bin/env bash

# Look for the AWS CLI
if test ! $(which aws); then
  echo "Installing AWS CLI..."
  brew install awscli
fi

# Look for the AWS configuration
if ! grep -q aws_access_key_id ~/.aws/config; then
  if ! grep -q aws_access_key_id ~/.aws/credentials; then
    echo 'AWS configuration was not found. Running "aws configure"...'
    aws configure
  fi
fi

Then in all of your scripts simply put:

$SWOODFORD_AWS_DIRECTORY/_pre_command.sh

Or something more fancy (but that I think isn't really necessary):

$SWOODFORD_AWS_DIRECTORY/_pre_command.sh || sh -c 'echo; echo "The file \"\$SWOODFORD_AWS_DIRECTORY/_pre_command.sh\" was not found or executable! Please see this page for help: https://github.com/swoodford/aws"'

And then you can put this in an install.sh script:

#!/usr/bin/env bash

# Look for Homebrew and install it if not found
if test ! $(which brew); then
  echo "Installing homebrew..."
  /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
fi

# Get the code
git clone https://github.com/swoodford/aws.git ~/.swoodford_aws

# Set this (probably not needed)
export SWOODFORD_AWS_DIRECTORY=~/.swoodford_aws

# Add SWOODFORD_AWS_DIRECTORY variable for reference
grep -q -i -F 'SWOODFORD_AWS_DIRECTORY' ~/.bash_profile || sh -c 'echo "SWOODFORD_AWS_DIRECTORY=\"~/.swoodford_aws"\" >> ~/.bash_profile'

# Add SWOODFORD_AWS_DIRECTORY to the PATH
grep -q -i -F 'PATH="~/.swoodford_aws' ~/.bash_profile || sh -c 'echo "PATH=\"~/.swoodford_aws:\$PATH\"" >> ~/.bash_profile'

Then you can tell users "To simply install these scripts run the following:
sh -c "$(curl -fsSL https://raw.githubusercontent.com/swoodford/aws/master/install.sh)" > /dev/null"