How can AWS CloudFormation help?

[cfndsl]

How can AWS CloudFormation help?

[domdivakaruni]

@ebekker @kornypoet @stevenjack we are working on a solution. Stay tuned!

[ebekker]

Is this request (for machine-digestible schema) still being tracked?

I think the current schema is a good start, however it is not complete when compared to the details defined in the User Guide (for example it doesn't offer some details such as "Update Requires" (Replacement | No interruption) and some of the details don't appear to be consistent with the User Guide or actual CFN implementation, for example, see this issue.

[howech]

I have several ideas, here are the first couple that spring to mind.

First, cfndsl maintains a yaml representation of the currently accepted set of language features that CloudFormation supports. If you look in the code, its in lib/cfndsl/aws_types.yaml. This file has been painstakingly put together by manually scraping through the CloudFormation template reference guide. About 90% of the issues that users have had with this project over the last couple of years was that AWS changed the language and cfndsl was slow at keeping up. If AWS were able to provide a machine digestable description of the CloudFormation template language, cfndsl could potentially kept up to date almost automatically.

My next suggestion does not directly relate to cfndsl, but I am certain that there are many cfndsl users who would agree. It is often something of a crap shoot to determine exactly what is going to happen when you update a stack. Sure, it is reasonably well documented down at the property level that changing the instance type of an instance will cause it to reboot (for ebs backed) or rebuild (for instance backed), etc. However, when presented with a large scale change to a large template (potentially a nested template...), it is difficult for a person to go through and work exactly what resources are going to be affected and i what ways. cfndsl does not currently capture the information about resource property update information (the above suggestion could help with that) if it did, cfndsl might be able to give users a better understanding about how update will affect their stacks. However, it would probably be much more useful to everyone if AWS CloudFormation had an api call that took all the parameters of an UpdateStack, but instead of doing anything, it just returned a description of all of the mayhem that would ensue if such an update were to take place.

Third, I realize that this isn't really AWS's fault, but at the moment, writing a template that is compatible with both AWS Cloudformation and Openstack Heat is extraordinarily difficult. For instance, Openstack does not use Base64 to encode the user data for its instances, so in an effort to make CloudFormation templates work at all, Heat has implemented its Fn::Base64 function as a pass-through. Unfortunately, there are places where it actually makes sense to base64 encode things, as it makes it so you dont have to worry about string escaping a payload that has to get transmitted over json. Accessing stack metadata does not seem to work the same way between CloudFormation and Heat, either (that may have improved with the Juno release - I havn't checked lately.) It would be nice if AWS and Openstack could get together and standardize a little bit, at least on a core set of features.

[cdandekar]

Thanks, @howech

[allinwonder]

• +1 on idea one. To avoid inconsistency, I had to fallback to use generate types and property. It will be good, AWS can provide an API to retrieve machine consumable resource tree with supported properties, also API calls to retrieve attributes and functions.
• about the second idea, The more practical solution is for CloudFormation to provide a dry-run option for stack create/update/delete, and in the dry-run mode it lists out all changes in the stacks without changing resources.

[cdandekar]

If AWS were able to provide a machine digestable description of the CloudFormation template language, cfndsl could potentially kept up to date almost automatically.

What kind of machine digestable description would that be?

[howech]

The bulk of the cloudformation language consists of Resources and Resource
Property Types. Resources are objects that are allowed to contain
properties that have name, value type and whether or not they are required
(sometimes the the required property rules are a little complicated as they
can be data dependent.) and how changes to a property on an update will
affect the resource. Here is an example taken from the aws user guide...

{
"Type" : "AWS::AutoScaling::AutoScalingGroup",
"Properties" : {
"AvailabilityZones
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-availabilityzones"
: [ String, ... ],
"Cooldown
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-cooldown"
: String,
"DesiredCapacity
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-desiredcapacity"
: String,
"HealthCheckGracePeriod
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-healthcheckgraceperiod"
: Integer,
"HealthCheckType
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-healthchecktype"
: String,
"InstanceId
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-instanceid"
: String,
"LaunchConfigurationName
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-launchconfigurationname"
: String,
"LoadBalancerNames
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-loadbalancernames"
: [ String, ... ],
"MaxSize http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-maxsize"
: String,
"MetricsCollection
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-metricscollection"
: [ MetricsCollection, ... ]
"MinSize http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-minsize"
: String,
"NotificationConfiguration
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-notificationconfiguration"
: NotificationConfiguration,
"PlacementGroup
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-placementgroup"
: String,
"Tags http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-tags"
: [ Auto Scaling Tag, ..., ],
"TerminationPolicies
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-termpolicy"
: [ String, ..., ],
"VPCZoneIdentifier
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-vpczoneidentifier"
: [ String, ... ]
}
}

NotificationConfiguration

An embedded property that configures an Auto Scaling group to send
notifications when specified events take place.

Required: No

Type: NotificationConfiguration
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-notificationconfiguration.html

Update requires: No interruption
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks.html#update-no-interrupt

Most of these are strings, lists of strings, ints, etc. Occasionally they
refer to a resource property type like MetricsCollection and
NotificationConfiguration. Resouce porpert types are similarly defined as
lists of property names, value types and whether or not it is required
(again, sometimes properties are only conditionally required. Here is an
example of a typical resource property type again taken from the cloud
formation user guide.
AWS CloudFormation AutoScaling Block Device Mapping Property Type

{
"DeviceName http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-launchconfig-blockdev-mapping.html#cfn-as-launchconfig-blockdev-mapping-devicename"
: String,
"Ebs http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-launchconfig-blockdev-mapping.html#cfn-as-launchconfig-blockdev-mapping-ebs"
: AutoScaling EBS Block Device,
"NoDevice http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-launchconfig-blockdev-mapping.html#cfn-as-launchconfig-blockdev-mapping-nodevice"
: Boolean,
"VirtualName
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-launchconfig-blockdev-mapping.html#cfn-as-launchconfig-blockdev-mapping-virtualname"
: String
}

Properties
DeviceName

The name of the device within Amazon EC2.

Required: Yes

Type: String
Ebs

The Amazon Elastic Block Store volume information.

Required: Conditional You can specify either VirtualName or Ebs, but not
both.

Type: AutoScaling EBS Block Device
http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-launchconfig-blockdev-template.html
.
NoDevice

Suppresses the device mapping. If NoDevice is set to true for the root
device, the instance might fail the Amazon EC2 health check. Auto Scaling
launches a replacement instance if the instance fails the health check.

Required: No

Type: Boolean
VirtualName

The name of the virtual device. The name must be in the form ephemeral_X_
where X is a number starting from zero (0), for example, ephemeral0.

Required: Conditional You can specify either VirtualName or Ebs, but not
both.

Type: String

If AWS could provide a javascript representation of what is shown in the
user guide to document the resource and resource types, cfndsl could use it
to adjust what it accepts as valid resource names, properties and resource
property types. For intsance, the above resource and resource property type
might appear as follows in a json document:

{ "Resources": [ ...
{ "Type": "AWS::AutoScaling::AutoScalingGroup",
"Properties": {
"AvailabilityZones": {"Type": ["String"],
"Required": false, "UpdateRequires": "NoInterruption" },
"Cooldown": {"Type": "String", "Required":
false, "UpdateRequires": "NoInterruption" },
...
"MetricsCollection": {"Type":
["MetricsCollection"], "Required": false, "UpdateRequires":
"NoInterruption" },
...
"NotivicationConfiguraion": { "Type":
"NotificationConfiguration", "Required": false, "UpdateRequires":
"NoInterruption" },
...
}
},
...
],

"ResourcePropertyTypes": { ...
"AWSCloudFormationAutoScalingBlockDeviceMapping": {
"DeviceName": {"Type": "String", "Required": true },
"EBS": {"Type": "AutoScalingEBSBlockDevice", "Required":
"Conditional" },
"NoDevice": {"Type": "Boolean", "Required": false },
"VirtualName": {"Type": "String", "Required": "Conditional" },
},
...
}
}

The language resources and resource properties that cfndsl currently
accepts is driven by a similar data structure that the developers have
pulled together manually by reading through the cloudformation user guide.
The manual process was fairly tedious and error prone, but also it quickly
grows stale as AWS adds more features and makes them accessible through
cloudformation.. If cfndsl were able to get to a currently updated list of
resources and resource property types similar to the above, it could keep
itself up to date with the latest allowed resources.

Also, note that currently cfndsl has no notion of properties being required
or conditional (as the data was just TOO tedious to gather manually). With
the above representation cfndsl would better be able to understand what it
takes to make a valid template, so it could throw error messages on
template compilation. Also, a future feature could be developed that would
allow users to explore what kinds of havoc a particular update might cause
their existing stack as cfndsl could be made aware of the implications of
updates made to various properties on the resources.

Thanks,

--Chris

On Thu, Feb 19, 2015 at 9:47 AM, Chetan Dandekar [email protected]
wrote:

If AWS were able to provide a machine digestable description of the
CloudFormation template language, cfndsl could potentially kept up to date
almost automatically.

What kind of machine digestable description would that be?

—
Reply to this email directly or view it on GitHub
#75 (comment).

[cdandekar]

Thanks @howech.

If AWS could provide a javascript representation of what is shown in the
user guide to document the resource and resource types, cfndsl could use it
to adjust what it accepts as valid resource names, properties and resource
property types.

Do you think a JSON schema (json-schema.org) based representation would work?

[howech]

It would take a little more work, but I think that that might work.

On Thu, Feb 19, 2015 at 11:43 AM, Chetan Dandekar [email protected]
wrote:

Thanks @howech https://github.com/howech.

If AWS could provide a javascript representation of what is shown in the
user guide to document the resource and resource types, cfndsl could use it
to adjust what it accepts as valid resource names, properties and resource
property types.

Do you think a JSON schema (json-schema.org) based representation would
work?

—
Reply to this email directly or view it on GitHub
#75 (comment).

[stevenjack]

I think a json schema based version would be great, that would provide most
of the validation of required fields, pattern matching and expected nodes.
On Thu, 19 Feb 2015 at 18:25, Chris Howe [email protected] wrote:

It would take a little more work, but I think that that might work.

On Thu, Feb 19, 2015 at 11:43 AM, Chetan Dandekar <
[email protected]>
wrote:

Thanks @howech https://github.com/howech.

If AWS could provide a javascript representation of what is shown in the
user guide to document the resource and resource types, cfndsl could use
it
to adjust what it accepts as valid resource names, properties and
resource
property types.

Do you think a JSON schema (json-schema.org) based representation would
work?

—
Reply to this email directly or view it on GitHub
#75 (comment).

—
Reply to this email directly or view it on GitHub
#75 (comment).

[howech]

Excellent explanation, Steven.

Just to clarify, when I said "more work", I of course meant more work on the cfndsl side but a json schema should be fine.

[allinwonder]

• +1 JSON format
• +1 include "Update requires" field, it is good to identify properties that requires replacement
• there are some conditional properties in CFN that are hard to represent in the JSON data, for example
  • AutoScalingGroup
    • InstanceId
      Required: Conditional. You must specify this property if you don't specify the LaunchConfigurationName property.

any idea on how to present this conditional property in the returned JSON?

[yumpy]

Is there a problem using the Eclipse Schema "CloudFormationV1.schema" referenced in #49?

http://vstoolkit.amazonwebservices.com/CloudFormationSchema/CloudFormationV1.schema

[stevenjack]

@ebekker I'm not sure if @cdandekar is still following this or if they have any updates on if they're thinking about providing any sort of schema apart from the one that's currently available above.. @cdandekar can you shed any light on this?

[webdevwilson]

@ebekker From a cursory glance, it looks like that would work. The inconsistencies could be an issue, but at least we would have a broader contributor base for fixing issues. Though eclipse in the title scares me a little as the eclipse foundation guidelines have been known to slow things down.

[kornypoet]

@cdandekar is there any new information regarding this issue?

[cmaxwellau]

Having a published Cloudformation schema is on the Cloudformation service team's roadmap, however I'm not in a position to talk about possible release dates.

[kornypoet]

Moving to #257

[ebekker]

Everyone interested in this should check out and provided feedback at #264