Feature Request: Support for "AWS::CloudFormation::Init" and "AWS::CloudFormation::Authentication" Types

[cfndsl]

Feature Request: Support for "AWS::CloudFormation::Init" and "AWS::CloudFormation::Authentication" Types

[howech]

These two types are substantially different from the rest of the resource types, because they live in a resource's metadata.

cfndsl currently does support metadata. For example, the following

# Example cribbed partially from aws cloudformation user guide
CloudFormation {
  Description "Example of using Metadata"

  Parameter("BucketName") {
    Type "String"
  }

  Parameter("S3User") {
    Type "String"
  }

  Instance("WebServer") {
    DependsOn "BucketPolicy"
    Metadata("AWS::CloudFormation::Authentication", {
               "S3AccessCreds" => {
                 "type" => "S3",
                 "accessKeyId" => Ref("CfnKeys"),
                 "secretKey" => FnGetAtt( "CfnKeys", "SecretAccessKey" )
               }
             })

    Metadata("AWS::CloudFormation::Init",
             {
               'config' => {
                 "packages" => { "yum" => { "httpd" => [] } },
                 "files" => {
                   "/var/www/html/index.html" => {
                     "source" => 
                       FnFormat( "http://s3.amazonaws.com/%{BucketName}/index.html"),               
                     "mode" => "000400",
                     "owner" => "apache",
                     "group" => "apache",
                     "authentication" => "S3AccessCreds"
                   }
                 },
                 "services" => {
                   "sysvinit" => {
                     "httpd" => { 
                       "enabled" => "true", 
                       "ensureRunning" => "true" }
                   }
                 }
               }
             })
    InstanceType "m1.small"
  }
  BucketPolicy("BucketPolicy") {
    Bucket Ref("BucketName")
    PolicyDocument({
                     "key" => "value" 
                   })
  }

  AccessKey("CfnKeys") {
    UserName Ref("S3User")
  }
}

Produces this:

{
   "Parameters" : {
      "BucketName" : {
         "Type" : "String"
      },
      "S3User" : {
         "Type" : "String"
      }
   },
   "Resources" : {
      "BucketPolicy" : {
         "Type" : "AWS::S3::BucketPolicy",
         "Properties" : {
            "Bucket" : {
               "Ref" : "BucketName"
            },
            "PolicyDocument" : {
               "key" : "value"
            }
         }
      },
      "CfnKeys" : {
         "Type" : "AWS::IAM::AccessKey",
         "Properties" : {
            "UserName" : {
               "Ref" : "S3User"
            }
         }
      },
      "WebServer" : {
         "DependsOn" : "BucketPolicy",
         "Type" : "AWS::EC2::Instance",
         "Properties" : {
            "InstanceType" : "m1.small"
         },
         "Metadata" : {
            "AWS::CloudFormation::Authentication" : {
               "S3AccessCreds" : {
                  "accessKeyId" : {
                     "Ref" : "CfnKeys"
                  },
                  "type" : "S3",
                  "secretKey" : {
                     "Fn::GetAtt" : [
                        "CfnKeys",
                        "SecretAccessKey"
                     ]
                  }
               }
            },
            "AWS::CloudFormation::Init" : {
               "config" : {
                  "packages" : {
                     "yum" : {
                        "httpd" : []
                     }
                  },
                  "services" : {
                     "sysvinit" : {
                        "httpd" : {
                           "ensureRunning" : "true",
                           "enabled" : "true"
                        }
                     }
                  },
                  "files" : {
                     "/var/www/html/index.html" : {
                        "owner" : "apache",
                        "source" : {
                           "Fn::Join" : [
                              "",
                              [
                                 "http://s3.amazonaws.com/",
                                 {
                                    "Ref" : "BucketName"
                                 },
                                 "/index.html"
                              ]
                           ]
                        },
                        "group" : "apache",
                        "mode" : "000400",
                        "authentication" : "S3AccessCreds"
                     }
                  }
               }
            }
         }
      }
   },
   "AWSTemplateFormatVersion" : "2010-09-09",
   "Description" : "Example of using Metadata"
}

II'l try to come up with better ways to do this, though.

[howech]

A question now would be, what should cfndsl support for CloudFormation::Init and CloudFormation::Authentication look like. Perhaps it would look something like this following template:

CloudFormation {
  Description "Example of using Metadata"

  Parameter("BucketName") {
    Type "String"
  }

  Parameter("S3User") {
    Type "String"
  }

  Instance("WebServer") {
    DependsOn "BucketPolicy"

    CloudFormation_Authentication("S3AccessCreds") {
      type "S3"
      accessKeyId Ref("CfnKeys")
      secretKey FnGetAtt( "CfnKeys", "SecretAccessKey" )
      # or perhaps even for these last two have a shortcut
      awsKey("CfnKeys")
    }

    CloudFormation_Init {
      config {
         package("yum", "httpd")
         file("/var/www/html/index.html") {
           source FnFormat( "http://s3.amazonaws.com/%{BucketName}/index.html"),               
           mode "000400",
           owner "apache",
           group "apache",
           authentication "S3AccessCreds"
         }
         service("sysvinit", "httpd") {
           enabled true
           ensureRunning true
         }

    |
    InstanceType "m1.small"
  }
  BucketPolicy("BucketPolicy") {
    Bucket Ref("BucketName")
    PolicyDocument({
                     "key" => "value" 
                   })
  }

  AccessKey("CfnKeys") {
    UserName Ref("S3User")
  }
}

Here, the methods CloudFormation_Authentication and CloudFormation_Init would be special methods that would create entries into the Metadata, sort of like, but somewhat different from Propterty methods like InstanceType. There would be a handful of new types (MetadataAuthentication, InitConfig, InitPackages, InitFiles, ...) Some of these new types would be dictionary types types instead of Array types, so this would mean writing some new metacoding architecture to handle the new types of types. I think that all of this can be done, but it is a resonably big chunk of work.

[allinwonder]

Hi Chris,

Your solution on both of the types are workable. But in order to maintain the consistency of type construction and being able to closely reflect similar structure to CloudFormation template. I prefer the following use case

  Instance("WebServer") {
    DependsOn "BucketPolicy"

    Metadata {
      Resource('S3AccessCreds'){
        Type 'AWS::CloudFormation::Authentication'
        ServiceType :S3
        AccessKeyId Ref("CfnKeys")
        AecretKey FnGetAtt( "CfnKeys", "SecretAccessKey" )
        Buckets(Ref("Bucket1"), Ref("Bucket2"))
      }
    }

    Metadata {
      AnnonymousResource{
        Type 'AWS::CloudFormation::Init'
        ConfigSet('set1', ['config1', 'config2'])
        Config('config1') {
          package("yum", "httpd")
          file("/var/www/html/index.html") {
           source FnFormat( "http://s3.amazonaws.com/%{BucketName}/index.html"),               
           mode "000400",
           owner "apache",
           group "apache",
           authentication "S3AccessCreds"
          }
          service("sysvinit", "httpd") {
           enabled true
           ensureRunning true
          }
        }
        # config2 goes into here similar to config 1
      }
    }
    InstanceType "m1.small"
  }

[howech]

I am afraid that I do not agree. Neither CloudFormation::Init nor CloudFormation::Authentication objects are structured like Resources, and calling them Resources would cloud the issue. Let me show you what I mean. Here is a snippet of template from the AWS CloudFormation Samples (http://aws.amazon.com/code/3233063344774450) from the S3Bucket_SourceAuth.template:

"Resources": {
...
 "WebServer": {  
      "Type": "AWS::EC2::Instance",
      "DependsOn" : "BucketPolicy",
      "Metadata" : {

        "AWS::CloudFormation::Init" : {
          "config" : {
            "packages" : {
              "yum" : {
                "httpd"        : []
              }
            },

            "sources" : {
                "/var/www/html" : { "Fn::Join" : ["", ["http://", { "Ref" : "Buc
ketName" }, ".s3.amazonaws.com/website.tar.gz"]]}
            },

            "services" : {
              "sysvinit" : {  
                "httpd" : { "enabled" : "true", "ensureRunning" : "true" }
              }
            }
          }
        },


        "AWS::CloudFormation::Authentication" : {
          "S3AccessCreds" : {
            "type" : "S3",
            "accessKeyId" : { "Ref" : "CfnKeys" },
            "secretKey" : {"Fn::GetAtt": ["CfnKeys", "SecretAccessKey"]},
            "buckets" : [ { "Ref" : "BucketName" } ]
          }
        }

      },
      "Properties": {
        "ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" }, { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] },
        "InstanceType"   : { "Ref" : "InstanceType" },
        "SecurityGroups" : [ {"Ref" : "WebServerSecurityGroup"} ],
        "KeyName"        : { "Ref" : "KeyName" },
        "UserData"       : { "Fn::Base64" : { "Fn::Join" : ["", [
          "#!/bin/bash\n",
          "yum update -y aws-cfn-bootstrap\n",

          "# Install application\n",
          "/opt/aws/bin/cfn-init -s ", { "Ref" : "AWS::StackId" }, " -r WebServer ",
          "    --region ", { "Ref" : "AWS::Region" }, "\n",

          "# All is well so signal success\n",
          "/opt/aws/bin/cfn-signal -e $? '", { "Ref" : "WaitHandle" }, "'\n"
        ]]}}        
      }
    },

}

Here "WebServer is a Resource - it lives directly in the resources container object, and it has the format

name: {
  Type: AWS::blah::blah
  Properties: {
     foo: bar
  }
}

The CloudFormation::Init and CloudFormation::Authentication sections live inside the Metadata on this resource. Even within that, they differ substantially from Resource objects:

 Metadata:
   AWS::CloudFormation::Type: {
     contents
   }

There is no "Type" section, no Properties section, nor any other of the trappings of a resource. Neither one of these creates a resource that you can use Ref or Fn::GetAtt on, they do not DependsOn anything (other than the implicit relationship they have with the Resource their metadata is on) nor can other resources DependsOn them.

Basically, the Metadata section of a resource is just a bib blob of json that gets associated with the created resource in the cloud. You can actually put anything you like in them, and get it out at runtime with a call to cfn-get-metadata. The other cfn-init scripts treat two sections of the metadata special - AWS::CloudFormation::Init and AWS::CloudFormation::Authentication.

[allinwonder]

Hi Chris,

I can see your point, and I agree that both of them are not directly attached to resources root. Now, I don't think they should be represented by Resource. So I scratch my previous idea.

Though they are not classified as resource, I think they are some kind of types that used in Metadata. And I still want them to be modeled rather than represented by a Hash.

So I come up a second idea that these two types has a capability to present themselves in Metadata. Please see the following use case for example

s3_access_creds = AWSCloudFormationAuthentication('S3AccessCreds'){
  Type 'S3'
  AccessKeyId Ref("CfnKeys")
  AecretKey FnGetAtt( "CfnKeys", "SecretAccessKey" )
  Buckets(Ref("Bucket1"), Ref("Bucket2"))
}

cfn_init = AWSCloudFormationInit {
  ConfigSet('set1', ['config1', 'config2'])
  Config('config1') {
    package("yum", "httpd")
    file("/var/www/html/index.html") {
      source FnFormat( "http://s3.amazonaws.com/%{BucketName}/index.html")
      mode "000400"
      owner "apache"
      group "apache"
      authentication "S3AccessCreds"
    }
    service("sysvinit", "httpd") {
      enabled true
      ensureRunning true
    }
  }
  #...
}

Metadata(s3_access_creds.to_metadata)
Metadata(cfn_init.to_metadata)