skymill/cumulus

Cumulus is a deployment suite used to deploy and manage environments built with AWS CloudFormation

Cumulus Deployment Suite

Cumulus

Cumulus Bundle Handler

Cumulus is a deployment suite used to deploy and manage environments built with AWS CloudFormation. Cumulus will help you bundle your code and configuration and unpack the bundle to new instances on CloudFormation.

Documentation

The Cumulus documentation is hosted at http://cumulus-ds.readthedocs.org/.

Installation

Cumulus consists of two parts, cumulus which is used to manage the software bundling and deployment and the cumulus-bundle-handler which handles the software installation on the target servers.

Installing cumulus

Install Cumulus via PyPI:

pip install cumulus

cumulus will now be available as a global command.

Installing cumulus-bundle-handler

Install Cumulus Bundle Handler via PyPI:

pip install cumulus-bundle-handler

cumulus-bundle-handler will now be available as a global command.

Attribution

This project is written and maintained by Sebastian Dahlgren (GitHub | Twitter | LinkedIn).

Cumulus development is supported by Skymill Solutions.

License

APACHE LICENSE 2.0 Copyright 2013-2014 Skymill Solutions

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

cumulus's Issues

Path prefix in bundles

Investigate best way for Cumulus to handle deploy paths.

Proposals:

• Included in bundle source (eg git)
• Added to every configured bundle path in cumulus.conf
• Using path rewrites in cumulus.conf (r//jenkins/workspace/onesonydev//var/www/wordpress)

Read the bucket name from configuration in CF template

Read the bucket name from configuration in CF template.

"/etc/cumulus/metadata.conf" : {
  "content" : { "Fn::Join" : ["", [
    "[metadata]\n",
    "aws_access_key_id: ", { "Ref" : "WebServerKeys" }, "\n",
    "aws_secret_access_key: ", {"Fn::GetAtt": ["WebServerKeys", "SecretAccessKey"]}, "\n",
    "region: ", {"Ref" : "AWS::Region"}, "\n",
    "s3_bundles_bucket: se.skymill.bundles\n",
    "stack: ", { "Ref" : "AWS::StackName" }, "\n",
    "bundle_type: webserver\n",
    "version: ", { "Ref" : "Version" }, "\n"
  ]]},
  "mode"  : "000644",
  "owner" : "root",
  "group" : "root"
},

Wait until stack is done updating/creating

Wait until stack is done updating/creating

Prefixes for prefixed files is not removed in bundle

Prefixes for prefixed files is not removed in bundle process. Must be fixed.

Investigate potential error with bundle deployment

Investigate potential error with bundle deployment.@danieljensen reported that wrong bundle may be fetched under some circumstances.

Get rid of Cumulus metadata.conf and make the bundle handler self-contained

Get rid of Cumulus metadata.conf and make the bundle handle self-contained. This will avoid sync issues in CloudFormation between metadata.conf and cfn-hup execution.

Current directory is added to bundle

The current directory is added (but empty) in the bundle.

[sebastian ~/Downloads]$ tar -jxvf bundle-stage-0.5.0-SNAPSHOT-webserver.tar.bz2
x Users/sebastian/git/skymill/web/hosts/webserver/
x etc/
...

Implement list stacks feature

It would be great to be able to list stacks per environment from the command line.

Mismatch in metadata and cumulus_bundle_handler.py

Mismatch in metadata and cumulus_bundle_handler.py.

root@ip-10-34-223-17:/var/log# cat cumulus_bundle_handler.log
2013-09-25T10:06:17.255795 - Connecting to AWS S3
2013-09-25T10:06:17.255949 - Missing config option: No option 'aws-access-key-id' in section: 'metadata'

Cumulus CF namespace conflicts with some rules

Cumulus namespace Cumulus:: collides with some rules:

2013-09-24 13:10:45,315 - cumulus - lib.config_handler - INFO - Reading configuration from ./cumulus.conf
2013-09-24 13:10:45,316 - cumulus - lib.stack_manager - INFO - Ensuring stack stage with template webserver.json
2013-09-24 13:10:45,929 - cumulus - boto - ERROR - 400 Bad Request
2013-09-24 13:10:45,929 - cumulus - boto - ERROR - {"Error":{"Code":"ValidationError","Message":"Template format error: Parameter name 'Cumulus::BundleBucket' is non alphanumeric.","Type":"Sender"},"RequestId":"f76fcf4c-2509-11e3-bc2f-97594e18ce34"}
2013-09-24 13:10:45,929 - cumulus - lib.stack_manager - ERROR - ERROR - Boto exception: BotoServerError: 400 Bad Request
None
2013-09-24 13:10:45,929 - cumulus - lib.stack_manager - ERROR - Enable debug in manage.py to see more details

Clean up host on bundle update

Currently, when updating a host the new bundle will be unpacked on top of the previous bundle. This means that if file A only existed in the oldest bundle, it will still be on the host when the newer bundle has been extracted.

We should clean up old files before extracting the new bundle.

Remove unnecessary bundle-type in metadata

Remove from JSON template:

"bundle-type: webserver\n",

Custom parameters in CloudFormation

I need to be able to send custom parameters to my CloudFormation templates through the configuration.

2013-09-24 12:12:43,034 - cumulus - lib.config_handler - INFO - Reading configuration from ./cumulus.conf
2013-09-24 12:12:43,035 - cumulus - lib.stack_manager - INFO - Ensuring stack stage with template webserver.json
2013-09-24 12:12:43,631 - cumulus - boto - ERROR - 400 Bad Request
2013-09-24 12:12:43,631 - cumulus - boto - ERROR - {"Error":{"Code":"ValidationError","Message":"Template requires parameter: LastUpdated","Type":"Sender"},"RequestId":"dad67735-2501-11e3-932f-c374bd49adf7"}
2013-09-24 12:12:43,632 - cumulus - lib.stack_manager - ERROR - ERROR - Boto exception: BotoServerError: 400 Bad Request
None
2013-09-24 12:12:43,632 - cumulus - lib.stack_manager - ERROR - Enable debug in manage.py to see more details

Enhance status output when waiting for stack change to complete

Enhance status output when waiting for stack change to complete

Initial stack creation fails when using --deploy-without-bundling

The stack creation fails if you run --deploy-without-bundling without having a bundle uploaded to S3 and no prior stacks running.

The deploy script will fail to fetch the bundle from S3 and hence the stack creation will also report failure.

Set environment version as input parameter

Let Cumulus get the environment version as a parameter (--version) that overrides the version specified in the config file.

Exception when building non-configured bundle

When trying to build a bundle that does not have any config the following exception appear:

2013-09-24 12:09:19,688 - cumulus - lib.bundler - INFO - Building bundle database
Traceback (most recent call last):
  File "../cumulus/cumulus/cumulus", line 11, in <module>
    lib.main()
  File "/Users/sebastian/git/skymill/cumulus/cumulus/lib/__init__.py", line 59, in main
    bundler.build_bundles()
  File "/Users/sebastian/git/skymill/cumulus/cumulus/lib/bundler.py", line 18, in build_bundles
    config_handler.get_bundle_paths(bundle))))
TypeError

My config at the time:

[environment: stage]
access-key-id: SECRET
secret-access-key: SECRET
bucket: se.skymill.bundles
region: eu-west-1
stacks: stage
bundles: webserver, database
version: 0.5.0-SNAPSHOT

[environment: production]
access-key-id: SECRET
secret-access-key: SECRET
bucket: se.skymill.bundles
region: eu-west-1
stacks: production
bundles: webserver, database
version: 0.4.0

[stack: stage]
template: /Users/sebastian/git/skymill/web/cf-templates/webserver.json
disable-rollback: true

[stack: production]
template: /Users/sebastian/git/skymill/web/cf-templates/webserver.json
disable-rollback: false

[bundle: webserver]
paths: /Users/sebastian/git/skymill/web/hosts/webserver

Remove skymill reference from JSON template

Remove skymill reference from JSON template

Bundler should add a generated metadata file to the bundle

Bundler should add a generated metadata file to the bundle. E.g. in /etc/cumulus-cloud-tools/metadata.conf.

Provide an example CloudFormation template

Provide an example CloudFormation template

config_handler is too large, split into submodules

The config_handler.py is too large. It would make sense to split it into smaller submodules, e.g. command line, configuration files etc.

Remove unnecessary stack name in metadata

Remove stack name from JSON template

"stack: ", { "Ref" : "AWS::StackName" }, "\n",

Update host script must be supplied

Update host script must be supplied somehow so that the instances knows how to fetch and extract bundles.

Specify config file location as input parameter

Since the Cumulus config most often will be part of checked out project code, specifying the config location as a parameter could be useful to prevent adding extra file moving steps to a deploy flow.

Write template validation

Write template validation for stacks

Symbolic links should be dereferenced in bundles

Symbolic links should be dereferenced in bundles when packed to resolve potential problems with broken links on instances.

Bundle type missing in Cumulus metadata

Bundle type missing in Cumulus metadata

Ensure that boto is available for cumulus bundle handler

Ensure that boto is available for cumulus bundle handler. The stack status should be failed if boto is missing.

Boto 505 Version not supported

Occasionally we get 505 Version not supported when creating, updating stacks or validating templates in CloudFormation. The CloudFormation team has responded that they can't fix this and that serving the template through S3 should resolve the issue, according to the related boto issue.

Uploading to S3 seems a bit to cumbersome to me (at least in order to be generic). I suggest that we implement a simply retry functionality.

Update `README.md` with proper docs

Update README.md with proper docs

Create shortcut for both bundling and deploying

It would be nice to have an option to both bundle and deploy with one command.

Write option to not remove old files in Cumulus bundle handler

Write option to not remove old files in Cumulus bundle handler. A command line option would suffice.

Expand ~ in config template & bundle paths

Expand ~ in config template & bundle paths

Filter events when creating/updating/deleting stacks

Currently the user will see old events when e.g. deleting a stack. We could work us around that by filtering event statuses depending on the action take. So if we're deleting a stack we'll only accept statuses like DELETE_IN_PROGRESS, DELETE_FAILED or DELETE_COMPLETE.

This is an example of the current output:

2013-10-27 12:37:48,057 - cumulus - INFO - Reading configuration from ./cumulus.conf
This will DELETE all stacks in the environment. This action cannot be undone. Are you sure you want to do continue? [N/y] y
2013-10-27 12:37:49,086 - cumulus - INFO - Deleting stack sdtest
2013-10-27 12:37:50,719 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - DELETE_IN_PROGRESS
2013-10-27 12:37:50,719 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - UPDATE_COMPLETE
2013-10-27 12:37:50,719 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - UPDATE_COMPLETE_CLEANUP_IN_PROGRESS
2013-10-27 12:37:50,719 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::LaunchConfiguration - UPDATE_COMPLETE
2013-10-27 12:37:50,719 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - UPDATE_IN_PROGRESS
2013-10-27 12:37:50,719 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - UPDATE_COMPLETE
2013-10-27 12:37:50,719 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - UPDATE_COMPLETE_CLEANUP_IN_PROGRESS
2013-10-27 12:37:50,719 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::LaunchConfiguration - UPDATE_COMPLETE
2013-10-27 12:37:50,719 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - UPDATE_IN_PROGRESS
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - UPDATE_COMPLETE
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - UPDATE_COMPLETE_CLEANUP_IN_PROGRESS
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::LaunchConfiguration - UPDATE_COMPLETE
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - UPDATE_IN_PROGRESS
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - CREATE_COMPLETE
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitCondition - CREATE_COMPLETE
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitCondition - CREATE_IN_PROGRESS
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitCondition - CREATE_IN_PROGRESS
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::AutoScalingGroup - CREATE_COMPLETE
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::AutoScalingGroup - CREATE_IN_PROGRESS
2013-10-27 12:37:50,720 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::AutoScalingGroup - CREATE_IN_PROGRESS
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::LaunchConfiguration - CREATE_COMPLETE
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::LaunchConfiguration - CREATE_IN_PROGRESS
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::LaunchConfiguration - CREATE_IN_PROGRESS
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::EC2::SecurityGroup - CREATE_COMPLETE
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::EC2::SecurityGroup - CREATE_IN_PROGRESS
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::IAM::AccessKey - CREATE_COMPLETE
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::IAM::AccessKey - CREATE_IN_PROGRESS
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::IAM::AccessKey - CREATE_IN_PROGRESS
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::IAM::User - CREATE_COMPLETE
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::IAM::User - CREATE_IN_PROGRESS
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitConditionHandle - CREATE_COMPLETE
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitConditionHandle - CREATE_IN_PROGRESS
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::IAM::User - CREATE_IN_PROGRESS
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitConditionHandle - CREATE_IN_PROGRESS
2013-10-27 12:37:50,721 - cumulus - INFO - Stack sdtest - AWS::EC2::SecurityGroup - CREATE_IN_PROGRESS
2013-10-27 12:37:50,722 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - CREATE_IN_PROGRESS
2013-10-27 12:38:07,761 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitCondition - DELETE_COMPLETE
2013-10-27 12:38:07,761 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitCondition - DELETE_IN_PROGRESS

Ask before delete when running --undeploy

Ask before delete when running --undeploy

Write proper documentation of the project

Write proper documentation of the project

Make it possible to delete running stacks

Make it possible to delete running stacks

Remove duplicate event rows

It is fairly common with duplicate event rows when creating stacks. This is natural as they are duplicate in the AWS Console as well (one with and one without the Physical ID set). We should either remove the duplicates or display the differences between them.

Current example output:

[sebastian ~/tmp/cumulus-test]$ ../../git/skymill/cumulus/cumulus/cumulus -e stage --deploy --version 1
2013-10-28 11:37:07,496 - cumulus - INFO - Reading configuration from ./cumulus.conf
2013-10-28 11:37:07,497 - cumulus - INFO - Building bundle webserver
2013-10-28 11:37:07,497 - cumulus - DEBUG - Bundle paths: /Users/sebastian/git/skymill/web/hosts/webserver
2013-10-28 11:37:09,070 - cumulus - INFO - Wrote bundle to ./target/bundle-stage-1-webserver.tar.bz2
2013-10-28 11:37:09,797 - cumulus - INFO - Starting upload of bundle-stage-1-webserver.tar.bz2
2013-10-28 11:37:18,335 - cumulus - INFO - Completed upload of bundle-stage-1-webserver.tar.bz2
2013-10-28 11:37:18,721 - cumulus - INFO - Uploading the cumulus_bundle_handler.py script
2013-10-28 11:37:18,812 - cumulus - INFO - Ensuring stack sdtest with template cloudformation-template-example.json
2013-10-28 11:37:19,260 - cumulus - DEBUG - Creating new stack with version 1
2013-10-28 11:37:21,636 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::Stack - CREATE_IN_PROGRESS
2013-10-28 11:37:32,806 - cumulus - INFO - Stack sdtest - AWS::IAM::AccessKey - CREATE_IN_PROGRESS
2013-10-28 11:37:32,806 - cumulus - INFO - Stack sdtest - AWS::IAM::User - CREATE_COMPLETE
2013-10-28 11:37:32,806 - cumulus - INFO - Stack sdtest - AWS::IAM::User - CREATE_IN_PROGRESS
2013-10-28 11:37:32,806 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitConditionHandle - CREATE_COMPLETE
2013-10-28 11:37:32,806 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitConditionHandle - CREATE_IN_PROGRESS
2013-10-28 11:37:32,806 - cumulus - INFO - Stack sdtest - AWS::EC2::SecurityGroup - CREATE_IN_PROGRESS
2013-10-28 11:37:32,806 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitConditionHandle - CREATE_IN_PROGRESS
2013-10-28 11:37:32,807 - cumulus - INFO - Stack sdtest - AWS::IAM::User - CREATE_IN_PROGRESS
2013-10-28 11:37:38,420 - cumulus - INFO - Stack sdtest - AWS::IAM::AccessKey - CREATE_COMPLETE
2013-10-28 11:37:38,420 - cumulus - INFO - Stack sdtest - AWS::IAM::AccessKey - CREATE_IN_PROGRESS
2013-10-28 11:37:49,783 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::LaunchConfiguration - CREATE_COMPLETE
2013-10-28 11:37:49,783 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::LaunchConfiguration - CREATE_IN_PROGRESS
2013-10-28 11:37:49,783 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::LaunchConfiguration - CREATE_IN_PROGRESS
2013-10-28 11:37:49,784 - cumulus - INFO - Stack sdtest - AWS::EC2::SecurityGroup - CREATE_COMPLETE
2013-10-28 11:37:49,784 - cumulus - INFO - Stack sdtest - AWS::EC2::SecurityGroup - CREATE_IN_PROGRESS
2013-10-28 11:37:55,479 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::AutoScalingGroup - CREATE_IN_PROGRESS
2013-10-28 11:37:55,479 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::AutoScalingGroup - CREATE_IN_PROGRESS
2013-10-28 11:39:16,263 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitCondition - CREATE_IN_PROGRESS
2013-10-28 11:39:16,263 - cumulus - INFO - Stack sdtest - AWS::CloudFormation::WaitCondition - CREATE_IN_PROGRESS
2013-10-28 11:39:16,263 - cumulus - INFO - Stack sdtest - AWS::AutoScaling::AutoScalingGroup - CREATE_COMPLETE

Add function for listing stack events on command line

Add function for listing stack events on command line

Running --deploy on existing stack fails

Running Cumulus with --deploy on a pre-existing stack causes it to abort instead of updating the stack

Cumulus bundle handler should use Python logging

Cumulus bundle handler should use Python logging module rather than stdout

Version should be per stack, not global

Version should be per stack, not globally defined

Always redeploy SNAPSHOT versions

Versions under current development should always be redeployed when updating a stack. This should be controlled via a configuration parameter in the config file. The parameter should be a regular expression.

Write hooks for Cumulus deployments

It would be great to have hooks when building bundles and deploying with Cumulus. I could think of the following hooks initially:

• Pre-bundling
• Post-bundling
• Pre-deploy
• Post-deploy

Errors should be thrown upwards

Currently some parts of Cumulus uses sys.exit() when a severe error has been encountered. This should be avoided and the errors should be thrown upwards and handled on a higher level.

The functions used for hooks, deployment etc should all return some kind of success status when completed.

Make it possible to environment prefix whole directories

Make it possible to environment prefix whole directories, not only files.

Error handling stack delete status

When the delete action of a stack/environment is complete Cumulus fails to check the delete status resulting in a stack trace. The stack is still removed successfully.

Output:

2013-10-23 11:44:45,884 - cumulus - lib.deployment_manager - INFO - Stack onesonydev - AWS::AutoScaling::LaunchConfiguration - DELETE_COMPLETE
2013-10-23 11:44:45,884 - cumulus - lib.deployment_manager - INFO - Stack onesonydev - AWS::AutoScaling::LaunchConfiguration - DELETE_IN_PROGRESS
2013-10-23 11:44:45,885 - cumulus - lib.deployment_manager - INFO - Stack onesonydev - AWS::AutoScaling::AutoScalingGroup - DELETE_COMPLETE
Traceback (most recent call last):
 File "./cumulus", line 11, in <module>
   lib.main()
 File "/var/lib/jenkins/onesonydev/cumulus/cumulus/lib/__init__.py", line 69, in main
   deployment_manager.undeploy()
 File "/var/lib/jenkins/onesonydev/cumulus/cumulus/lib/deployment_manager.py", line 44, in undeploy
   _delete_stack(stack)
 File "/var/lib/jenkins/onesonydev/cumulus/cumulus/lib/deployment_manager.py", line 135, in _delete_stack
   _wait_for_stack_complete(stack)
 File "/var/lib/jenkins/onesonydev/cumulus/cumulus/lib/deployment_manager.py", line 226, in _wait_for_stack_complete
   if stack.stack_status in complete_statuses:
AttributeError: 'NoneType' object has no attribute 'stack_status'

Cumulus bundle handler should support both start and kill scripts in init.d

Cumulus bundle handler should support both start and kill scripts in init.d, so that we can start and stop services pre and post bundle extraction.

Remove __name__ from logging output

Remove __name__ from logging output

Implement support for `disable_rollback` in stacks

Implement support for disable_rollback in stacks