skymill/cumulus

Cumulus is a deployment suite used to deploy and manage environments built with AWS CloudFormation

cumulus's Issues

Write hooks for Cumulus deployments

It would be great to have hooks when building bundles and deploying with Cumulus. I could think of the following hooks initially:

• Pre-bundling
• Post-bundling
• Pre-deploy
• Post-deploy

Filter events when creating/updating/deleting stacks

Currently the user will see old events when e.g. deleting a stack. We could work us around that by filtering event statuses depending on the action take. So if we're deleting a stack we'll only accept statuses like DELETE_IN_PROGRESS, DELETE_FAILED or DELETE_COMPLETE.

This is an example of the current output:

2013-10-27 12:37:48,057 - cumulus - INFO - Reading configuration from ./cumulus.conf
This will DELETE all stacks in the environment. This action cannot be undone. Are you sure you want to do continue? [N/y] y
2013-10-27 12:37:49,086 - cumulus - INFO - Deleting stack sdtest
2013-10-27 12:37:50,719 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - DELETE_IN_PROGRESS
2013-10-27 12:37:50,719 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - UPDATE_COMPLETE
2013-10-27 12:37:50,719 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - UPDATE_COMPLETE_CLEANUP_IN_PROGRESS
2013-10-27 12:37:50,719 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::LaunchConfiguration - UPDATE_COMPLETE
2013-10-27 12:37:50,719 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - UPDATE_IN_PROGRESS
2013-10-27 12:37:50,719 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - UPDATE_COMPLETE
2013-10-27 12:37:50,719 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - UPDATE_COMPLETE_CLEANUP_IN_PROGRESS
2013-10-27 12:37:50,719 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::LaunchConfiguration - UPDATE_COMPLETE
2013-10-27 12:37:50,719 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - UPDATE_IN_PROGRESS
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - UPDATE_COMPLETE
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - UPDATE_COMPLETE_CLEANUP_IN_PROGRESS
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::LaunchConfiguration - UPDATE_COMPLETE
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - UPDATE_IN_PROGRESS
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - CREATE_COMPLETE
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitCondition - CREATE_COMPLETE
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitCondition - CREATE_IN_PROGRESS
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitCondition - CREATE_IN_PROGRESS
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::AutoScalingGroup - CREATE_COMPLETE
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::AutoScalingGroup - CREATE_IN_PROGRESS
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::AutoScalingGroup - CREATE_IN_PROGRESS
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::LaunchConfiguration - CREATE_COMPLETE
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::LaunchConfiguration - CREATE_IN_PROGRESS
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::LaunchConfiguration - CREATE_IN_PROGRESS
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::EC2::SecurityGroup - CREATE_COMPLETE
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::EC2::SecurityGroup - CREATE_IN_PROGRESS
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::IAM::AccessKey - CREATE_COMPLETE
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::IAM::AccessKey - CREATE_IN_PROGRESS
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::IAM::AccessKey - CREATE_IN_PROGRESS
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::IAM::User - CREATE_COMPLETE
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::IAM::User - CREATE_IN_PROGRESS
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitConditionHandle - CREATE_COMPLETE
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitConditionHandle - CREATE_IN_PROGRESS
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::IAM::User - CREATE_IN_PROGRESS
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitConditionHandle - CREATE_IN_PROGRESS
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::EC2::SecurityGroup - CREATE_IN_PROGRESS
2013-10-27 12:37:50,722 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - CREATE_IN_PROGRESS
2013-10-27 12:38:07,761 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitCondition - DELETE_COMPLETE
2013-10-27 12:38:07,761 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitCondition - DELETE_IN_PROGRESS

Cumulus bundle handler should use Python logging

Cumulus bundle handler should use Python logging module rather than stdout

Remove duplicate event rows

It is fairly common with duplicate event rows when creating stacks. This is natural as they are duplicate in the AWS Console as well (one with and one without the Physical ID set). We should either remove the duplicates or display the differences between them.

Current example output:

[sebastian ~/tmp/cumulus-test]$ ../../git/skymill/cumulus/cumulus/cumulus -e stage --deploy --version 1
2013-10-28 11:37:07,496 - cumulus - INFO - Reading configuration from ./cumulus.conf
2013-10-28 11:37:07,497 - cumulus - INFO - Building bundle webserver
2013-10-28 11:37:07,497 - cumulus - DEBUG - Bundle paths: /Users/sebastian/git/skymill/web/hosts/webserver
2013-10-28 11:37:09,070 - cumulus - INFO - Wrote bundle to ./target/bundle-stage-1-webserver.tar.bz2
2013-10-28 11:37:09,797 - cumulus - INFO - Starting upload of bundle-stage-1-webserver.tar.bz2
2013-10-28 11:37:18,335 - cumulus - INFO - Completed upload of bundle-stage-1-webserver.tar.bz2
2013-10-28 11:37:18,721 - cumulus - INFO - Uploading the cumulus_bundle_handler.py script
2013-10-28 11:37:18,812 - cumulus - INFO - Ensuring stack sdtest with template cloudformation-template-example.json
2013-10-28 11:37:19,260 - cumulus - DEBUG - Creating new stack with version 1
2013-10-28 11:37:21,636 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - CREATE_IN_PROGRESS
2013-10-28 11:37:32,806 - cumulus - INFO - Stack sdtest - AWS::IAM::AccessKey - CREATE_IN_PROGRESS
2013-10-28 11:37:32,806 - cumulus - INFO - Stack sdtest - AWS::IAM::User - CREATE_COMPLETE
2013-10-28 11:37:32,806 - cumulus - INFO - Stack sdtest - AWS::IAM::User - CREATE_IN_PROGRESS
2013-10-28 11:37:32,806 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitConditionHandle - CREATE_COMPLETE
2013-10-28 11:37:32,806 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitConditionHandle - CREATE_IN_PROGRESS
2013-10-28 11:37:32,806 - cumulus - INFO - Stack sdtest - AWS::EC2::SecurityGroup - CREATE_IN_PROGRESS
2013-10-28 11:37:32,806 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitConditionHandle - CREATE_IN_PROGRESS
2013-10-28 11:37:32,807 - cumulus - INFO - Stack sdtest - AWS::IAM::User - CREATE_IN_PROGRESS
2013-10-28 11:37:38,420 - cumulus - INFO - Stack sdtest - AWS::IAM::AccessKey - CREATE_COMPLETE
2013-10-28 11:37:38,420 - cumulus - INFO - Stack sdtest - AWS::IAM::AccessKey - CREATE_IN_PROGRESS
2013-10-28 11:37:49,783 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::LaunchConfiguration - CREATE_COMPLETE
2013-10-28 11:37:49,783 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::LaunchConfiguration - CREATE_IN_PROGRESS
2013-10-28 11:37:49,783 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::LaunchConfiguration - CREATE_IN_PROGRESS
2013-10-28 11:37:49,784 - cumulus - INFO - Stack sdtest - AWS::EC2::SecurityGroup - CREATE_COMPLETE
2013-10-28 11:37:49,784 - cumulus - INFO - Stack sdtest - AWS::EC2::SecurityGroup - CREATE_IN_PROGRESS
2013-10-28 11:37:55,479 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::AutoScalingGroup - CREATE_IN_PROGRESS
2013-10-28 11:37:55,479 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::AutoScalingGroup - CREATE_IN_PROGRESS
2013-10-28 11:39:16,263 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitCondition - CREATE_IN_PROGRESS
2013-10-28 11:39:16,263 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitCondition - CREATE_IN_PROGRESS
2013-10-28 11:39:16,263 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::AutoScalingGroup - CREATE_COMPLETE

Cumulus bundle handler should support both start and kill scripts in init.d

Cumulus bundle handler should support both start and kill scripts in init.d, so that we can start and stop services pre and post bundle extraction.

config_handler is too large, split into submodules

The config_handler.py is too large. It would make sense to split it into smaller submodules, e.g. command line, configuration files etc.

Wait until stack is done updating/creating

Wait until stack is done updating/creating

Current directory is added to bundle

The current directory is added (but empty) in the bundle.

[sebastian ~/Downloads]$ tar -jxvf bundle-stage-0.5.0-SNAPSHOT-webserver.tar.bz2
x Users/sebastian/git/skymill/web/hosts/webserver/
x etc/
...

Investigate potential error with bundle deployment

Investigate potential error with bundle deployment.@danieljensen reported that wrong bundle may be fetched under some circumstances.

Remove skymill reference from JSON template

Remove skymill reference from JSON template

Write template validation

Write template validation for stacks

Create shortcut for both bundling and deploying

It would be nice to have an option to both bundle and deploy with one command.

Always redeploy SNAPSHOT versions

Versions under current development should always be redeployed when updating a stack. This should be controlled via a configuration parameter in the config file. The parameter should be a regular expression.

Get rid of Cumulus metadata.conf and make the bundle handler self-contained

Get rid of Cumulus metadata.conf and make the bundle handle self-contained. This will avoid sync issues in CloudFormation between metadata.conf and cfn-hup execution.

Symbolic links should be dereferenced in bundles

Symbolic links should be dereferenced in bundles when packed to resolve potential problems with broken links on instances.

Bundler should add a generated metadata file to the bundle

Bundler should add a generated metadata file to the bundle. E.g. in /etc/cumulus-cloud-tools/metadata.conf.

Path prefix in bundles

Investigate best way for Cumulus to handle deploy paths.

Proposals:

• Included in bundle source (eg git)
• Added to every configured bundle path in cumulus.conf
• Using path rewrites in cumulus.conf (r//jenkins/workspace/onesonydev//var/www/wordpress)

Initial stack creation fails when using --deploy-without-bundling

The stack creation fails if you run --deploy-without-bundling without having a bundle uploaded to S3 and no prior stacks running.

The deploy script will fail to fetch the bundle from S3 and hence the stack creation will also report failure.

Boto 505 Version not supported

Occasionally we get 505 Version not supported when creating, updating stacks or validating templates in CloudFormation. The CloudFormation team has responded that they can't fix this and that serving the template through S3 should resolve the issue, according to the related boto issue.

Uploading to S3 seems a bit to cumbersome to me (at least in order to be generic). I suggest that we implement a simply retry functionality.

Errors should be thrown upwards

Currently some parts of Cumulus uses sys.exit() when a severe error has been encountered. This should be avoided and the errors should be thrown upwards and handled on a higher level.

The functions used for hooks, deployment etc should all return some kind of success status when completed.

Read the bucket name from configuration in CF template

Read the bucket name from configuration in CF template.

"/etc/cumulus/metadata.conf" : {
  "content" : { "Fn::Join" : ["", [
    "[metadata]\n",
    "aws_access_key_id: ", { "Ref" : "WebServerKeys" }, "\n",
    "aws_secret_access_key: ", {"Fn::GetAtt": ["WebServerKeys", "SecretAccessKey"]}, "\n",
    "region: ", {"Ref" : "AWS::Region"}, "\n",
    "s3_bundles_bucket: se.skymill.bundles\n",
    "stack: ", { "Ref" : "AWS::StackName" }, "\n",
    "bundle_type: webserver\n",
    "version: ", { "Ref" : "Version" }, "\n"
  ]]},
  "mode"  : "000644",
  "owner" : "root",
  "group" : "root"
},

Cumulus CF namespace conflicts with some rules

Cumulus namespace Cumulus:: collides with some rules:

2013-09-24 13:10:45,315 - cumulus - lib.config_handler - INFO - Reading configuration from ./cumulus.conf
2013-09-24 13:10:45,316 - cumulus - lib.stack_manager - INFO - Ensuring stack stage with template webserver.json
2013-09-24 13:10:45,929 - cumulus - boto - ERROR - 400 Bad Request
2013-09-24 13:10:45,929 - cumulus - boto - ERROR - {"Error":{"Code":"ValidationError","Message":"Template format error: Parameter name 'Cumulus::BundleBucket' is non alphanumeric.","Type":"Sender"},"RequestId":"f76fcf4c-2509-11e3-bc2f-97594e18ce34"}
2013-09-24 13:10:45,929 - cumulus - lib.stack_manager - ERROR - ERROR - Boto exception: BotoServerError: 400 Bad Request
None
2013-09-24 13:10:45,929 - cumulus - lib.stack_manager - ERROR - Enable debug in manage.py to see more details

Write proper documentation of the project

Write proper documentation of the project

Remove unnecessary bundle-type in metadata

Remove from JSON template:

"bundle-type: webserver\n",

Ensure that boto is available for cumulus bundle handler

Ensure that boto is available for cumulus bundle handler. The stack status should be failed if boto is missing.

Make it possible to delete running stacks

Make it possible to delete running stacks

Write option to not remove old files in Cumulus bundle handler

Write option to not remove old files in Cumulus bundle handler. A command line option would suffice.

Error handling stack delete status

When the delete action of a stack/environment is complete Cumulus fails to check the delete status resulting in a stack trace. The stack is still removed successfully.

Output:

2013-10-23 11:44:45,884 - cumulus - lib.deployment_manager - INFO - Stack onesonydev - AWS::AutoScaling::LaunchConfiguration - DELETE_COMPLETE
2013-10-23 11:44:45,884 - cumulus - lib.deployment_manager - INFO - Stack onesonydev - AWS::AutoScaling::LaunchConfiguration - DELETE_IN_PROGRESS
2013-10-23 11:44:45,885 - cumulus - lib.deployment_manager - INFO - Stack onesonydev - AWS::AutoScaling::AutoScalingGroup - DELETE_COMPLETE
Traceback (most recent call last):
 File "./cumulus", line 11, in <module>
   lib.main()
 File "/var/lib/jenkins/onesonydev/cumulus/cumulus/lib/__init__.py", line 69, in main
   deployment_manager.undeploy()
 File "/var/lib/jenkins/onesonydev/cumulus/cumulus/lib/deployment_manager.py", line 44, in undeploy
   _delete_stack(stack)
 File "/var/lib/jenkins/onesonydev/cumulus/cumulus/lib/deployment_manager.py", line 135, in _delete_stack
   _wait_for_stack_complete(stack)
 File "/var/lib/jenkins/onesonydev/cumulus/cumulus/lib/deployment_manager.py", line 226, in _wait_for_stack_complete
   if stack.stack_status in complete_statuses:
AttributeError: 'NoneType' object has no attribute 'stack_status'

Clean up host on bundle update

Currently, when updating a host the new bundle will be unpacked on top of the previous bundle. This means that if file A only existed in the oldest bundle, it will still be on the host when the newer bundle has been extracted.

We should clean up old files before extracting the new bundle.

Remove __name__ from logging output

Remove __name__ from logging output

Update `README.md` with proper docs

Update README.md with proper docs

Implement support for `disable_rollback` in stacks

Implement support for disable_rollback in stacks

Update host script must be supplied

Update host script must be supplied somehow so that the instances knows how to fetch and extract bundles.

Remove unnecessary stack name in metadata

Remove stack name from JSON template

"stack: ", { "Ref" : "AWS::StackName" }, "\n",

Bundle type missing in Cumulus metadata

Bundle type missing in Cumulus metadata

Add function for listing stack events on command line

Add function for listing stack events on command line

Make it possible to environment prefix whole directories

Make it possible to environment prefix whole directories, not only files.

Ask before delete when running --undeploy

Ask before delete when running --undeploy

Version should be per stack, not global

Version should be per stack, not globally defined

Enhance status output when waiting for stack change to complete

Enhance status output when waiting for stack change to complete

Exception when building non-configured bundle

When trying to build a bundle that does not have any config the following exception appear:

2013-09-24 12:09:19,688 - cumulus - lib.bundler - INFO - Building bundle database
Traceback (most recent call last):
  File "../cumulus/cumulus/cumulus", line 11, in <module>
    lib.main()
  File "/Users/sebastian/git/skymill/cumulus/cumulus/lib/__init__.py", line 59, in main
    bundler.build_bundles()
  File "/Users/sebastian/git/skymill/cumulus/cumulus/lib/bundler.py", line 18, in build_bundles
    config_handler.get_bundle_paths(bundle))))
TypeError

My config at the time:

[environment: stage]
access-key-id: SECRET
secret-access-key: SECRET
bucket: se.skymill.bundles
region: eu-west-1
stacks: stage
bundles: webserver, database
version: 0.5.0-SNAPSHOT

[environment: production]
access-key-id: SECRET
secret-access-key: SECRET
bucket: se.skymill.bundles
region: eu-west-1
stacks: production
bundles: webserver, database
version: 0.4.0

[stack: stage]
template: /Users/sebastian/git/skymill/web/cf-templates/webserver.json
disable-rollback: true

[stack: production]
template: /Users/sebastian/git/skymill/web/cf-templates/webserver.json
disable-rollback: false

[bundle: webserver]
paths: /Users/sebastian/git/skymill/web/hosts/webserver

Running --deploy on existing stack fails

Running Cumulus with --deploy on a pre-existing stack causes it to abort instead of updating the stack

Set environment version as input parameter

Let Cumulus get the environment version as a parameter (--version) that overrides the version specified in the config file.

Expand ~ in config template & bundle paths

Expand ~ in config template & bundle paths

Specify config file location as input parameter

Since the Cumulus config most often will be part of checked out project code, specifying the config location as a parameter could be useful to prevent adding extra file moving steps to a deploy flow.

Custom parameters in CloudFormation

I need to be able to send custom parameters to my CloudFormation templates through the configuration.

2013-09-24 12:12:43,034 - cumulus - lib.config_handler - INFO - Reading configuration from ./cumulus.conf
2013-09-24 12:12:43,035 - cumulus - lib.stack_manager - INFO - Ensuring stack stage with template webserver.json
2013-09-24 12:12:43,631 - cumulus - boto - ERROR - 400 Bad Request
2013-09-24 12:12:43,631 - cumulus - boto - ERROR - {"Error":{"Code":"ValidationError","Message":"Template requires parameter: LastUpdated","Type":"Sender"},"RequestId":"dad67735-2501-11e3-932f-c374bd49adf7"}
2013-09-24 12:12:43,632 - cumulus - lib.stack_manager - ERROR - ERROR - Boto exception: BotoServerError: 400 Bad Request
None
2013-09-24 12:12:43,632 - cumulus - lib.stack_manager - ERROR - Enable debug in manage.py to see more details

Implement list stacks feature

It would be great to be able to list stacks per environment from the command line.

Mismatch in metadata and cumulus_bundle_handler.py

Mismatch in metadata and cumulus_bundle_handler.py.

root@ip-10-34-223-17:/var/log# cat cumulus_bundle_handler.log
2013-09-25T10:06:17.255795 - Connecting to AWS S3
2013-09-25T10:06:17.255949 - Missing config option: No option 'aws-access-key-id' in section: 'metadata'

Provide an example CloudFormation template

Provide an example CloudFormation template

Prefixes for prefixed files is not removed in bundle

Prefixes for prefixed files is not removed in bundle process. Must be fixed.