Giter Site home page Giter Site logo

aws-cloudformation / cloudformation-resource-schema Goto Github PK

View Code? Open in Web Editor NEW
93.0 28.0 38.0 213 KB

The CloudFormation Resource Schema defines the shape and semantic for resources provisioned by CloudFormation. It is used by provider developers using the CloudFormation RPDK.

License: Apache License 2.0

Java 100.00%
aws aws-cloudformation cloudformation

cloudformation-resource-schema's Introduction

AWS CloudFormation Resource Schema

Build Status

This document describes the Resource Provider Definition Schema which is a meta-schema that extends draft-07 of JSON Schema to define a validating document against which resource schemas can be authored.

Examples

Numerous examples exist in this repository to help you understand various shape and semantic definition models you can apply to your own resource definitions.

Defining Resources

Overview

The meta-schema which controls and validates your resource type definition is called the Resource Provider Definition Schema. It is fully compliant with draft-07 of JSON Schema and many IDEs including IntelliJ, PyCharm and Visual Studio Code come with built-in or plugin-based support for code-completion and syntax validation while editing documents for JSON Schema compliance. Comprehensive documentation for JSON Schema exists and can answer many questions around correct usage.

To get started, you will author a specification for your resource type in a JSON document, which must be compliant with this meta-schema. To make authoring resource specifications simpler, we have constrained the scope of the full JSON Schema standard to apply opinions around how certain validations can be expressed and encourage consistent modelling for all resource schemas. These opinions are codified in the meta-schema and described in this document.

Resource Type Name

All resources MUST specify a typeName which adheres to the Regular Expression ^[a-zA-Z0-9]{2,64}::[a-zA-Z0-9]{2,64}::[a-zA-Z0-9]{2,64}$. This expression defines a 3-part namespace for your resource, with a suggested shape of Organization::Service::Resource. For example AWS::EC2::Instance or Initech::TPS::Report. This typeName is how you will address your resources for use in CloudFormation and other provisioning tools.

Resource Shape

The shape of your resource defines the properties for that resource and how they should be applied. This includes the type of each property, validation patterns or enums, and additional descriptive metadata such as documentation and example usage. Refer to the #/definitions/properties section of the meta-schema for the full set of supported properties you can use to describe your resource shape.

Resource Semantics

Certain properties of a resource are semantic and have special meaning when used in different contexts. For example, a property of a resource may be readOnly when read back for state changes - but can be specified in a settable context when used as the target of a $ref from a related resource. Because of this semantic difference in how this property metadata should be interpreted, certain aspects of the resource definition are applied to the parent resource definition, rather than at a property level. Those elements are;

  • primaryIdentifier: Must be either a single property, or a set of properties which can be used to uniquely identify the resource. If multiple properties are specified, these are treated as a composite key and combined into a single logical identifier. You would use this modelling to express contained identity (such as a named service within a container). This property can be independently provided as keys to a READ or DELETE request and MUST be supported as the only input to those operations. These properties are usually also marked as readOnlyProperties and MUST be returned from READ and LIST operations.
  • additionalIdentifiers: Each property listed in the additionalIdentifiers section must be able to be used to uniquely identify the resource. These properties can be independently provided as keys to a READ or DELETE request and MUST be supported as the only input to those operations. These properties are usually also marked as readOnlyProperties and MUST be returned from READ and LIST operations. A provider is not required to support additionalIdentifiers; doing so allows for other unique keys to be used to READ resources.
  • readOnlyProperties: A property in the readOnlyProperties list cannot be specified by the customer.
  • writeOnlyProperties: A property in the writeOnlyProperties cannot be returned in a READ or LIST request, and can be used to express things like passwords, secrets or other sensitive data.
  • createOnlyProperties: A property in the createOnlyProperties cannot be specified in an UPDATE request, and can only be specified in a CREATE request. Another way to think about this - these are properties which are 'write-once', such as the Engine property for an AWS::RDS::DBInstance and if you wish to change such a property on a live resource, you should replace that resource by creating a new instance of the resource and terminating the old one. This is the behaviour CloudFormation follows for all properties documented as 'Update Requires: Replacement'. An attempt to supply these properties to an UPDATE request will produce a runtime error from the handler.
  • deprecatedProperties: A property in the deprecatedProperties is not guaranteed to be present in the response from a READ request. These fields will still be accepted as input to CREATE and UPDATE requests however they may be ignored, or converted to new API forms when outbound service calls are made.
  • replacementStrategy: As mentioned above, changing a createOnlyProperty requires replacement of the resource by creating a new one and deleting the old one. The default CloudFormation replacement behavior is to create a new resource first, then delete the old resource, so as to avoid any downtime. However, some resources are singleton resources, meaning that only one can exist at a time. In this case, it is not possible to create a second resource first, so CloudFormation must Delete first and then Create. Specify either create_then_delete or delete_then_create. Default value is create_then_delete
  • taggable: [DEPRECATED] A boolean type property which defaults to true, indicating this resource type supports updatable tagging property. Otherwise, it indicates this resource type does not contain any updatable tagging properties.
  • tagging: An object type property that indicates whether this resource type supports AWS tags, tagging behavior, and what property is used to set tags:
    • taggable: A boolean flag indicating whether the resource type supports tagging.
    • tagOnCreate: A boolean flag indicating whether the resource type supports passing tags in the create API.
    • tagUpdatable: A boolean flag indicating whether the resource type can modify resouce's tags using update handler.
    • cloudFormationSystemTags: A boolean flag indicating whether the resource type create handler can apply aws prefixed tags, CloudFormation system tags.
    • tagProperty: A reference to the Tags property in the schema.
    • Examples:
      • "tagging": {
            "taggable": false
        }
      • "tagging": {
            "taggable": true,
            "tagOnCreate": true,
            "tagUpdatable": true,
            "cloudFormationSystemTags": true,
            "tagProperty": "/properties/Tags"
        }
  • propertyTransform: Is a map (Map<String, String>) with the keys being property paths and values being jsonata transformation functions (https://jsonata.org/). This property is used to avoid falsely drifted resources. If the handler transforms the input to the resource to an expected value a transform function can be defined for this property to avoid drift.

Application

When defining resource semantics like createOnlyProperties, primaryIdentifier you are expected to use a JSON Pointer to a property definition in the same resource document. Schemas you author can be checked with the CFN CLI validate command.

The following (truncated) example shows some of the semantic definitions for an AWS::S3::Bucket resource type;

{
    "$id": "aws-s3-bucket.json",
    "typeName": "AWS::S3::Bucket",
    "resourceLink": {
        "templateUri": "/s3/home?region=${awsRegion}&bucket=${BucketName}",
        "mappings": {
            "BucketName": "/BucketName"
        }
    },
    "definitions": {
        "NestedDefinitions" : {
              "type" : "object",
              "additionalProperties" : false,
              "properties" : {
                "ReturnData" : {
                  "type" : "boolean"
                },
                "Expression" : {
                  "type" : "string"
                }
          },
    },
    "properties": {
        "Arn": {
            "$ref": "aws.common.types.v1.json#/definitions/Arn"
        },
        "BucketName": {
            "type": "string"
        },
        "Id": {
            "type": "integer"
        },
        "NestedProperty": {
            "$ref": "#/definitions/NestedDefinitions"
        }
    },
    "createOnlyProperties": [
        "/properties/BucketName"
    ],
    "readOnlyProperties": [
        "/properties/Arn"
    ],
    "primaryIdentifier": [
        "/properties/BucketName"
    ],
    "additionalIdentifiers": [
        "/properties/Arn",
        "/properties/WebsiteURL"
    ],
    "propertyTransform": {
        "/properties/Id": "$abs(Id) $OR $power(Id, 2)",
        "/properties/NestedProperty/Expression": $join(["Prefix", Expression])
    }
}

Note: $OR is supported between 2 Jsontata functions or experessions. It is not supported as part of a string. Following use of $OR is not supported in propertyTransform: "/properties/e": '$join([e, "T $OR Y"])',

Relationships

Relationships between resources can be expressed through the use of the $ref keyword when defining a property schema. The use of the $ref keyword to establish relationships is described in JSON Schema documentation.

Example

The following example shows a property relationship between an AWS::EC2::Subnet.VpcId and an AWS::EC2::VPC.Id. The schema for the 'remote' type (AWS::EC2::VPC) is used to validate the content of the 'local' type (AWS::EC2::Subnet) and can be inferred as a dependency from the local to the remote type.

Setting the $id property to a remote location will make validation framework to pull dependencies expressed using relative $ref URIs from the remote hosts. In this example, VpcId property will be verified against the schema for AWS::EC2::VPC.Id hosted at https://schema.cloudformation.us-east-1.amazonaws.com/aws-ec2-vpc.json

{
    "$id": "https://schema.cloudformation.us-east-1.amazonaws.com/aws-ec2-subnet.json",
    "typeName": "AWS::EC2::Subnet",
    "definitions": { ... },
    "properties": {
        { ... }
        "VpcId": {
            "$ref": "aws-ec2-vpc.json#/properties/Id"
        }
    }
}
{
    "$id": "https://schema.cloudformation.us-east-1.amazonaws.com/aws-ec2-vpc.json",
    "typeName": "AWS::EC2::VPC",
    "definitions": { ... },
    "properties": {
        "Id": {
            "type": "string",
            "pattern": "$vpc-[0-9]{8,10}^"
        }
    }
}

Divergence From JSON Schema

Changes

We have taken an opinion on certain aspects of the core JSON Schema and introduced certain constrains and changes from the core schema. In the context of this project, we are not building arbitrary documents, but rather, defining a very specific shape and semantic for cloud resources.

  • readOnly: the readOnly field as defined in JSON Schema does not align with our determination that this is actually a restriction with semantic meaning. A property may be readOnly when specified for a particular resource (for example it's Arn), but when that same property is referenced (using $ref tokens) from a dependency, the dependency must be allowed to specify an input for that property, and as such, it is no longer readOnly in that context. The AWS CloudFormation Resource Schema uses the concept of readOnlyProperties for this mechanic.
  • writeOnly: see above

New Schema-Level Properties

insertionOrder

Array types can define a boolean insertionOrder, which specifies whether the order in which elements are specified should be honored when processing a diff between two sets of properties. If insertionOrder is true, then a change in order of the elements will constitute a diff. The default for insertionOrder is true.

Together with the uniqueItems property (which is native to JSON Schema), complex array types can be defined, as in the following table:

insertionOrder uniqueItems result
true false list
false false multiset
true true ordered set
false true set

arrayType

arrayType is used to specify the type of array and is only applicable for properties of type array. When set to AttributeList, it indicates that the array is used to represent a list of additional properties, and when set to Standard it indicates that the array consists of a list of values. The default for arrayType is Standard. For example, 'Standard' would be used for an array of Arn values, where the addition of the values themselves has significance. An example of using 'AttributeList' would be for a list of optional, and often defaulted, values that can be specified. For example, 'AttributeList' would be used for an array of TargetGroupAttributes for ELB where addition of the default values has no significance.

Constraints

  • $id: an $id property is not valid for a resource property.
  • $schema: a $schema property is not valid for a resource property.
  • if, then, else, not: these imperative constructs can lead to confusion both in authoring a resource definition, and for customers authoring a resource description against your schema. Also this construct is not widely supported by validation tools and is disallowed here.
  • propertyNames: use of propertyNames implies a set of properties without a defined shape and is disallowed. To constrain property names, use patternProperties statements with defined shapes.
  • additionalProperties use of additionalProperties is not valid for a resource property. Use patternProperties instead to define the shape and allowed values of extraneous keys.
  • properties and patternProperties it is not valid to use both properties and patternProperties together in the same shape, as a shape should not contain both defined and undefined values. In order to implement this, the set of undefined values should itself be a subshape.
  • items and additionalItems the items in an array may only have one schema and may not use a list of schemas, as an ordered tuple of different objects is confusing for both developers and customers. This should be expressed as key:value object pairs. Similarly, additionalItems is not allowed.
  • replacementStrategy: a replacementStrategy is not valid for a mutable resource that does not need replacement during an update.

handlers

The handlers section of the schema allows you to specify which CRUDL operations (create, read, update, delete, list) are available for your resource, as well as some additional metadata about each handler.

permissions

For each handler, you should define a list of API permissions required to perform the operation. Currently, this is used to generate IAM policy templates and is assumed to be AWS API permissions, but you may list 3rd party APIs here as well.

timeoutInMinutes

For each handler, you may define a timeoutInMinutes property, which defines the maximum timeout of the operation. This timeout is used by the invoker of the handler (such as CloudFormation) to stop listening and cancel the operation. Note that the handler may of course decide to timeout and return a failure prior to this max timeout period. Currently, this value is only used for Create, Update, and Delete handlers, while Read and List handlers are expected to return synchronously within 30 seconds.

License

This library is licensed under the Apache 2.0 License.

cloudformation-resource-schema's People

Contributors

akifrafique avatar ammokhov avatar anshikg avatar augustxiaodm avatar aygold92 avatar bcorman3 avatar cortez7 avatar elruwen avatar gsurbhi avatar malikatalla avatar marcuram avatar mattmcloudy avatar narmadn avatar nina-ctrlv avatar omkhegde avatar patmyron avatar rjlohan avatar rootulp avatar srujithpoondla03 avatar syldyer avatar tobywf avatar vladtsir avatar waela avatar wenyhu avatar xiwhuang avatar yueful27 avatar yunhao-jiao avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cloudformation-resource-schema's Issues

ResourceTypeSchema fails when loading resource definitions using 'oneOf', 'if' keywords

Error:

java.lang.ClassCastException: org.everit.json.schema.CombinedSchema$Builder cannot be cast to org.everit.json.schema.ObjectSchema$Builder
	at software.amazon.cloudformation.resource.ResourceTypeSchema.load(ResourceTypeSchema.java:110)

To reproduce create a schema file as follows:

valid-with-oneof.json:

{
  "typeName": "AWS::Test::TestModel",
  "description": "A test schema for unit tests.",
  "sourceUrl": "https://mycorp.com/my-repo.git",
  "properties": {
    "propertyA": {
      "type": "string"
    },
    "propertyB": {
      "type": "string"
    }
  },
  "oneOf": [
    { "required":  ["propertyA"]},
    { "required" : ["propertyB"]}
  ],
  "primaryIdentifier": [
    "/properties/propertyA"
  ],
  "additionalProperties": false
}
public void validSchema_withOneOf_shouldSucceed() {
        JSONObject resource = loadJSON("/valid-with-oneof.json");
        final ResourceTypeSchema schema = ResourceTypeSchema.load(resource);
}

Incomprehensible error message when using AWS::CloudFormation::ResourceVersion & AWS::CloudFormation::PublicTypeVersion

Hi,

We are using a StackSet to deploy our custom Public Type to all regions in tandem. For that, we are utilizing çAWS::CloudFormation::ResourceVersion, AWS::CloudFormation::ResourceDefaultVersion, AWS::CloudFormation::Publisher, and AWS::CloudFormation::PublicTypeVersion. They are listed in the same order as specified here, and each item DependsOn the item that precedes it.

Unfortunately, we are unable to pass the tests, and get this error:

ResourceLogicalId:SomeProductPublicTypeVersion, ResourceType:AWS::CloudFormation::PublicTypeVersion, ResourceStatusReason:Resource handler returned message: "Model validation failed (#: #: only 1 subschema matches out of 2) #: #: 2 subschemas matched instead of one (#)" (RequestToken: b8399ca3-0aec-572d-XXX-XXXX, HandlerErrorCode: InvalidRequest).

A few things are worth noting here:

  1. In the log files that are directly uploaded to our S3 (using the LogDeliveryBucket property of PublicTypeVersion), our custom type version seems to pass 100% of the tests. There is not indication of any errors.

  2. If we publish the type manually (through the cli, it's a multi-step, lengthy process), all goes fine and we can successfully publish the type (publicly). We are using the same SchemaHandlerPackage URL for both the manual flow, and the StackSet. The former works, the latter doesn't. The error message is close to useless given that the log files show no errors at all.

  3. The schema itself passes cfn validate

Please advise.

misleading property name: AWS::Logs::LogGroup.KmsKeyId

Hello,

I got problem when I use this template to deploy KMSKey, KMSAlias and CW LogGroup :

Resources:
  KMSKey:
    Type: 'AWS::KMS::Key'
    Properties: 
      Description: KMS for CloudWatch Log group
      Enabled: true
      KeyPolicy: 
        Version: '2012-10-17'
        Id: key-default-1
        Statement:
        - Sid: Enable IAM User Permissions
          Effect: Allow
          Principal:
            AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
          Action: kms:*
          Resource: '*'
        - Sid: Enable CWL Permissions
          Effect: Allow
          Principal:
            Service: !Sub logs.${AWS::Region}.amazonaws.com
          Action:
          - kms:Describe*
          - kms:Encrypt*
          - kms:Decrypt*
          - kms:ReEncrypt*
          - kms:ReEncrypt*
          Resource: '*'

  KMSAlias:  
    Type: 'AWS::KMS::Alias'
    Properties: 
      AliasName: alias/CloudWatchLogKMS
      TargetKeyId: !Ref KMSKey     
  LogMessage:
    Type: 'AWS::Logs::LogGroup'
    Properties: 
      LogGroupName: /var/log/messages
      RetentionInDays: 365
      KmsKeyId: !Ref KMSKey

Error message :
Model validation failed (#/KmsKeyId: failed validation constraint for keyword [pattern])

In the doc of "AWS::KMS::Key" resource, return value is key ID.
But "AWS::Logs::LogGroup" wait an ARN not a KeyID. The param himself if ambiguous "KmsKeyId" but in reality is KmsKeyArn

My workaround is to use "Fn::GetAtt" instead of Ref :
KmsKeyId: !GetAtt KMSKey.Arn

Regards,
Tcheksa

Obfuscated schema validation messages make resolution cumbersome

Troubleshooting schema validation errors is cumbersome and should be more customer focused.

While troubleshooting a resource being provisioned the builder is left trying to figure out a few things. Based on the scrubbed messaging the user has to:

  1. Determine what resource failed and its type
  2. Run describe type on the type specified
  3. Relate the error message, the properties of the resource, and the json schema definition to determine the correct pattern. When the definition names don't match property names this can become more difficult. When there are many layers of sub properties this can become more annoying. And when the value is a Ref/GetAtt to another resource its even more difficult to understand and translate what went wrong.

Example CloudFormation validation failure:
Model validation failed (#/KmsKeyId: failed validation constraint for keyword [pattern])

Example failure when calling the service itself (Lambda example):
1 validation error detected: Value 'NotAArn' at 'kMSKeyArn' failed to satisfy constraint: Member must satisfy regular expression pattern: (arn:(aws[a-zA-Z-]*)?:[a-z0-9-.]+:.*)|()

Obfuscated messaging was added by #40

I can understand not printing the value with secretsmanager dynamic references but we should at least be able to provide a path to the error and what is allowed (pattern, min/max length, etc.). This would eliminate so many of the above steps.

Also the layering of these error messages can make troubleshooting harder. This may be a documentation thing but took me a few times looking at the errors above to understand if the error was coming from CloudFormation or form the service itself. In this scenario the error coming from the service was more detailed and allowed me to resolve the issue quicker than the CloudFormation error but I'm not sure that is always the case.

How to identify resource attributes?

Hi there,

When reading the schema of any resource, I cannot figure out how to identify the resource attributes.

Take for example AWS::Cloud9::EnvironmentEC2, the AWS CloudFormation doc states that Arn and Name can be retrieved via Fn::GetAtt, however in the corresponding JSON schema (see below), these two fields are specified in the properties of the specification but there is no additional information that tag them as being available as attributes via Fn::GetAtt.

Are we supposed to continue to use the AWS CloudFormation Resource Specification in addition to these new JSON schemas? Or you guys will update the schemas to add the missing pieces? Or I am totally wrong about how we are supposed to leverage these schemas?

{
  "typeName" : "AWS::Cloud9::EnvironmentEC2",
  "description" : "Resource Type definition for AWS::Cloud9::EnvironmentEC2",
  "additionalProperties" : false,
  "properties" : {
    "Arn" : {
      "type" : "string"
    },
    "Name" : {
      "type" : "string"
    },
    "Repositories" : {
      "type" : "array",
      "uniqueItems" : false,
      "items" : {
        "$ref" : "#/definitions/Repository"
      }
    },
    "OwnerArn" : {
      "type" : "string"
    },
    "Description" : {
      "type" : "string"
    },
    "AutomaticStopTimeMinutes" : {
      "type" : "integer"
    },
    "SubnetId" : {
      "type" : "string"
    },
    "EnvironmentId" : {
      "type" : "string"
    },
    "InstanceType" : {
      "type" : "string"
    }
  },
  "definitions" : {
    "Repository" : {
      "type" : "object",
      "additionalProperties" : false,
      "properties" : {
        "PathComponent" : {
          "type" : "string"
        },
        "RepositoryUrl" : {
          "type" : "string"
        }
      },
      "required" : [ "RepositoryUrl", "PathComponent" ]
    }
  },
  "required" : [ "InstanceType" ],
  "readOnlyProperties" : [ "/properties/Arn", "properties/EnvirontmentId" ],
  "createOnlyProperties" : [ "/properties/AutomaticStopTimeMinutes", "/properties/OwnerArn", "/properties/Name", "/properties/InstanceType", "/properties/SubnetId", "/properties/Repositories", "/properties/Description" ],
  "primaryIdentifier" : [ "properties/EnvirontmentId" ],
  "additionalIdentifiers" : [ "/properties/Arn" ]
}

Add support for links to resources

Add an e.g. deepLink property to the schema that allows generating links to a resource given identifiers. This is useful for the console.

New property `handlerSchema` in `handlerDefinition` not in `provider.definition.schema.v1.json`

Recent updates to some CloudFormation resource schemas

  • AWS::NetworkManager::TransitGatewayRegistration
  • AWS::NimbleStudio::LaunchProfile
  • AWS::NimbleStudio::StreamingImage
  • AWS::NimbleStudio::StudioComponent

have added a new property, handlerSchema , to certain handlers, e.g.

    "list": {
      "handlerSchema": {
        "properties": {
          "GlobalNetworkId": {
            "$ref": "resource-schema.json#/properties/GlobalNetworkId"
          }
        },
        "required": ["GlobalNetworkId"]
      },
      "permissions": [
        "networkmanager:GetTransitGatewayRegistrations"
      ]
    },

This property has not been added to https://github.com/aws-cloudformation/cloudformation-resource-schema/blob/master/src/main/resources/schema/provider.definition.schema.v1.json, so validation of these resource schemas against the meta-schema are failing.

Serializer improvements

We should have a method that serializes to a string, as a lot of the time we take the JSONObject returned from the current method and immediately call toString. Small improvement but easy to add

MWAA::Environment fails to create: "#/SourceBucketArn: failed validation constraint for keyword [pattern]"

Hi,

I'm trying to create an MWAA Environment resource via. CloudFormation. When attempting to create the stack I receive an error:

Error: ROLLBACK_COMPLETE: ["The following resource(s) failed to create: [AirflowMWAA]. Rollback requested by user." "Properties validation failed for resource AirflowMWAA with message:\n#/SourceBucketArn: failed validation constraint for keyword [pattern]"]

Here's a snippet of my CF stack:

Description: airflow-head
Resources:
  AirflowMWAA:
    Type: AWS::MWAA::Environment
    Properties:
      ...
      SourceBucketArn: arn:aws:s3:::my-bucket-name
      ...
    DeletionPolicy: Delete

The bucket ARN is valid and is accepted if I create the env via. the MWAA wizard in the AWS console.
How can I fix this? The error is too terse to be useful.

Thanks,
Nigel.

Support complex permissions

Currently, the permissions model only supports action-based permissions with no additional scope. I propose the addition of a complex permissions set such as:

{
    "typeName": "Foo::Bar::Baz",
    ...
    "handlers": {
        "create": {
            "complexPermissions": [
                {
                    "Action": [
                        "s3:GetObject",
                        "s3:PutObject",
                        "s3:List*"
                    ],
                    "Resource": [
                        "arn:aws:s3:::limited*",
                        "arn:aws:s3:::otherlimited*"
                    ],
                    "Condition": [
                        "ForAnyValue:StringEquals": {
                            "aws:CalledVia": [
                                "dynamodb.amazonaws.com"
                            ]
                        }
                    ],
                    "Effect": "Allow"
                },
                {
                    "Action": [
                        "s3:GetObject",
                        "s3:PutObject",
                        "s3:List*"
                    ],
                    "NotResource": [
                        "arn:aws:s3:::limited*"
                    ],
                    "Effect": "Deny"
                },
                ...
            ]
        },
        ...
    }
}

Labels: enhancement

Support use of `deprecated` field for properties.

Resource properties should be able to be marked as "deprecated": true to indicate that their use on input operations (such as CREATE, UPDATE) will be accepted but possibly ignored, or will take no effect.

This field will also indicate that such properties may not be present on READ operations.

Or, we should add a deprecatedProperties semantic section for the same purpose. Need to decide which is most appropriate.

New property `relationshipRef` not in `provider.definition.schema.v1.json`

Recent updates to some CloudFormation resource schemas

  • AWS::ECS::Cluster
  • AWS::S3::MultiRegionAccessPoint
  • AWS::SSM::Association

have added a new property, relationshipRef , to certain properties, e.g.

        "Region": {
            "type": "object",
            "properties": {
                "Bucket": {
                    "type": "string",
                    "minLength": 3,
                    "maxLength": 63,
                    "pattern": "^[a-z0-9][a-z0-9//.//-]*[a-z0-9]$",
                    "relationshipRef": {
                        "typeName": "AWS::S3::Bucket",
                        "propertyPath": "/properties/BucketName"
                    }
                }, 
                "BucketAccountId": {
                    "type": "string",
                    "minLength": 12,
                    "maxLength": 12,
                    "pattern": "^[0-9]{12}$"
                }
            },
            "required": [
                "Bucket"
            ],
            "additionalProperties": false
        }

This property has not been added to https://github.com/aws-cloudformation/cloudformation-resource-schema/blob/master/src/main/resources/schema/provider.definition.schema.v1.json, so validation of these resource schemas against the meta-schema are failing.

Look at adding removedProperties to schema

Should consider deprecatedProperties as those which still work but will maybe not one day, where removedProperties are a strong declaration that the property is no longer valid for the type.

Allow additionalItems keyword into schema

We originally converted items to disallow mixed tuple validation of lists, and rather expected an array to allow only a single consistent type.

However, we also omitted the additionalItems keyword which can be used in certain validation cases to allow an array to validate a single type with a fixed set of allowed values. An example of this is the AWS::CloudFront::Distribution.DistributionConfig.CacheBehavior.AllowedMethods field which uses a set of array values as a wonky enum validation.

SEE: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-cloudfront-distribution-cachebehavior.html#cfn-cloudfront-distribution-cachebehavior-allowedmethods

This needs to be put into the schema;

"additionalItems": { 
    "type": "boolean"
}

Replace `identifiers` with `primaryIdentifiers` and `additionalIdentifiers`

Currently, we have a semantic schema for identifiers which allows a type to define more than one set of identifiers, where each set is either a single JSONPointer to a property, or an ordered list of JSONPointers to multiple properties, where a unique identifier is formed as a composite key.

Generally this is fine, but it would make usage clearer for some caller scenarios if a single identifier (single property, or composite key) was defined as the primary identifier. This identifier should globally, uniquely identify an instance of the modeled resource and can be relied on to never change for the life of a resource. This would also require that the CloudFormation registry ensure that primary identifiers cannot change for a published resource type once defined.

We also want to allow additional identifiers which can be used as inputs to READ operations, in order to cater for multiple external systems which may be relying on a different form of identifier, such as using an S3 Bucket ARN rather than an S3 BucketName. If a provider chose to support additionalIdentifiers, the requirement on those would be that a READ operation should be able to use those identifiers to fully describe a model, including its primary identifier. This is unchanged from the current schema contract.

additionalIdentifiers would be offered for improved user-experience for callers, and would not be subject to ongoing limitations. They can be re-ordered, removed, changed, etc for each published version of a resource type. As long as the READ handler for that type meets the contract, the additionalIdentifiers would be valid for that version.

The proposed schema change I have is;

"primaryIdentifier": {
    "description": "A required identifier which uniquely identifies an instance of this resource type. An identifier is a non-zero-length list of JSON pointers to properties that form a single key. An identifier can be a single or multiple properties to support composite-key identifiers.",
    "$ref": "#/definitions/jsonPointerArray"
},
"additionalIdentifiers": {
    "description": "An optional list of supplementary identifiers, each of which uniquely identifies an instance of this resource type. An identifier is a non-zero-length list of JSON pointers to properties that form a single key. An identifier can be a single or multiple properties to support composite-key identifiers.",
    "type": "array",
    "minItems": 1,
    "items": {
        "$ref": "#/definitions/jsonPointerArray"
    }
}

Add runtime schema validation for things that can't be checked by JSONSchema

  • a property can't be readOnly and writeOnly.
  • a property can't be readOnly and createOnly.
  • an identifier must be either createOnly or readOnly (note: this one is debatable, but CFN does not currently support updating the physicalResourceId)
  • ensure that jsonPointers map to actual properties

Update (3/16/2020)

Please add more as we come across them

PublicTypeVersion NOT actually incrementing public version (?)

Following up on the thread where I reported an issue (#129) which actually turned out to be an issue on the CFN side, I used a workaround suggested there by @ugudip for publishing a public type:

Hey SHxKM,
Thanks for the template. I don't think it makes any difference by changing the resource name you give, you are allowed to give any name as a logical Id. Your template looks good at first glance, but I will need to investigate on why it isn't working for you. I will get back to you shortly with my findings. Meanwhile, as a workaround, can you please try by giving TypeName and Type instead of Arn for PublicTypeVersion resource?

So I've used this template to register the custom resource in almost all regions:

Resources:
  MyTypePrivateTypeVersion: 
    Type: AWS::CloudFormation::ResourceVersion
    Properties:
      SchemaHandlerPackage: !Ref SchemaHandlerUrl
      TypeName: MyCompany::MyCategory::MyType
  MyTypeResourceDefaultVersion:    
    Type: AWS::CloudFormation::ResourceDefaultVersion
    DependsOn: MyTypePrivateTypeVersion
    Properties:
      TypeVersionArn: !Ref MyTypePrivateTypeVersion
  CustomTypePublisher:
    Type: AWS::CloudFormation::Publisher
    DependsOn: MyTypeResourceDefaultVersion
    Properties:
      AcceptTermsAndConditions: true
  MyTypePublicTypeVersion:     
    Type: AWS::CloudFormation::PublicTypeVersion
    DependsOn: CustomTypePublisher
    Properties:
      LogDeliveryBucket: my-s3-logs-bucket
      PublicVersionNumber: !Ref VersionToPublish
      Type: RESOURCE
      TypeName: MyCompany::MyCategory::MyType

The registration of version 1.0.0 was successful.

However, when I tried to update the version to 1.0.1, even though the operation was "successful", the version isn't really bumped.

  • The operation was marked as successful
  • The stack instance has a status of UPDATE_COMPLETE
  • under the StackSet the stack instance its status is CURRENT
  • In the Parameters part of the stack instance, the version is 1.0.1

Despite all this, almost 40 minutes after the "successful" publishing, the version showing in the public registry is still 1.0.0. It seems that there is no real sync whatsoever.

There is one detail that may be affecting this process. When I registered the types initially, I used:

aws cloudformation create-stack-instances --stack-set-name my-stack-set-name --accounts '["XXXXX"]'  --regions '["LIST OF REGIONS HERE"]' --operation-preferences FailureToleranceCount=0,MaxConcurrentCount=1,RegionConcurrencyType=PARALLEL

The StackSet parameters were:

VersionToPublish=1.0.1 # this may be the culprit 
SchemaHandlerUrl=s3://my-bucket/my-handler.zip

Notice the 1.0.1 as the parameter value. What this did: it published the public types successfully, but as version 1.0.0 - this is documented in PublicTypeVersion:

The first time you publish a type, AWS CloudFormation sets the version number to 1.0.0, regardless of the value you specify.

Any attempts to then update the template failed, or reported "No updates are to be performed" (because the template showed 1.0.1, even though the actual underlying resource version was still 1.0.0, so we were stuck. As a workaround, I've uploaded an identical .zip to my-handler.zip, and updated the stack-set changing only the handler URL to the new (yet identical zip):

aws cloudformation update-stack-set --stack-set-name my-stack-set-name --template-body "file://my-public-type-stackset.yaml" --region us-east-1 --parameters ParameterKey=VersionToPublish,ParameterValue=1.0.1 ParameterKey=SchemaHandlerUrl,ParameterValue=s3://my-bucket/my-handler-updated.zip

This operation was successful, but as I detailed above, it does not seem to actually publish the new version. What is the proper way to "force" all stack instances to update, given the same parameters?

"Sub-list" for resources with multiple primary identifiers

When a resource schema providers more than one property for primaryIdentifier, it may make sense to list the resources for a fixed value of one or more of those properties.

We're building an S3 bucket notification resource, to solve a long-standing problem. Each item in the notification configuration for a bucket is a separate resource. The primaryIdentifier is the combination of the bucket name and the notification configuration item id. The list handler handler will iterate over all buckets, returning all notification configuration items inside. But it's also sensible to think that a user could ask to list all notification resources for a fixed bucket name.

It seems likely that people will order their primary identifier list from general to specific, so may it could be automatic that if a list handler supports this "sub-listing" the user could provide the first value, the first and second, etc.

Incorrect JSON Schema draft URI

According to the spec, the draft-07 schema URI should begin with http:// and not https:// (schema ID does not necessarily mean the network location where the resource is stored).

So I believe the references should be changed to reflect this.

Validate $ref correctly

Issue

See also aws-cloudformation/cloudformation-cli#66

TL;DR: If we validate the schema as data, and our meta schema allows $ref, it doesn't work, as the "data" i.e. the schema is not further processed:

@Test
public void improperValidation() {
    final JSONObject definition = new JSONObject()
            .put(TYPE_NAME_KEY, EXAMPLE_TYPE_NAME)
            .put(DESCRIPTION_KEY, EXAMPLE_DESCRIPTION)
            .put(PROPERTIES_KEY, new JSONObject()
                .put("propertyA", new JSONObject()
                    .put("$ref", "#/garbage")))
            .put(PRIMARY_IDENTIFIER_KEY, Arrays.asList(EXAMPLE_PRIMARY_IDENTIFIER));

    validator.validateResourceDefinition(definition);
}

this passed, but the $ref is obviously garbage. The reason it "passes" is because we allow $refs:

https://github.com/aws-cloudformation/aws-cloudformation-resource-schema/blob/6a1ef45d612d7c90da1d77e71a9427a64cdd5237/src/main/resources/schema/provider.definition.schema.v1.json#L85-L87

however, all JSON schema mandates is:

"$ref": {
    "type": "string",
    "format": "uri-reference"
},

Solutions?

The obvious solution is to resolve all $refs before validation, and then it'd work. In Python, I'd use jsonref, or disallow circular references and simply inline all references, and disallow $ref.

It may be possible to use org.everit.json.schema.loader.SchemaLoader in some capacity. Instead of validating against draft-7 (the usual meta-schema), it may be possible to use our meta-schema instead. Since SchemaLoader handles $refs semantically, that ought to work. (I had planned to do something like this in the RPDK, where the library/dynamic nature makes it fairly straightforward to do.)

If in future we want to restrict $refs to the local or any hosted schemas, putting in our own code could be beneficial here in the long run.

Cross-resource constraints

We're building an S3 bucket notification resource, to solve a long-standing problem. Each item in the notification configuration for a bucket is a separate resource. However, the rules for bucket notifications says they can't overlap in prefix+events. So two resources can share a prefix if they subscribe to different events, or if they share events they can't share a prefix. If someone creates set of resources that violates these constraints, the S3 API call will return an error at some point in the process, which we'll surface to the user. But it got me thinking if these kinds of constraints could be expressed in the schema. It could be something like "uniquePropertySets": {"SomeNameForThis": [ '#/properties/SomeScalarProperty', '#/properties/SomeSetValuedProperty']}

What is the expected format of `readOnlyProperties`?

What is the expected format of readOnlyProperties? There are several examples in the current schemas, shown below. Since this format is an extension of JSON Schema, I'd have expected readOnlyProperties to match required, i.e. just having property names. Instead it has something which looks like a JSON Pointer, but in most cases isn't actually a valid reference.

Are all of the properties listed here supposed to be now accessible with Fn::GetAtt? The last word on this was in 2019 (#59): "supposed to be, but not quite working yet".

Prefixed with /properties/, nested with /

The /properties/ prefix implies that this is a JSON Pointer, but these aren't actually valid references, since the nested property isn't a child of the schema at that location.

E.g. AWS::RDS::DBInstance:

"readOnlyProperties": [
  "/properties/Endpoint/Address",
  "/properties/Endpoint/Port",
  "/properties/Endpoint/HostedZoneId",
  "/properties/DbiResourceId",
  "/properties/DBInstanceArn",
  "/properties/DBSystemId",
  "/properties/MasterUserSecret/SecretArn",
  "/properties/CertificateDetails/CAIdentifier",
  "/properties/CertificateDetails/ValidTill"
],

Prefixed with /properties/, nested with /, contains * wildcards

Also not a valid reference. E.g. AWS::WAFv2::RuleGroup:

"readOnlyProperties": [
  "/properties/Arn",
  "/properties/Id",
  "/properties/LabelNamespace",
  "/properties/AvailableLabels/*/Name",
  "/properties/ConsumedLabels/*/Name"
],

Prefixed with /properties/, nested with .

Also not a valid reference. E.g. AWS::ElastiCache::CacheCluster:

"readOnlyProperties": [
  "/properties/ConfigurationEndpoint.Address",
  "/properties/Id",
  "/properties/ConfigurationEndpoint.Port",
  "/properties/RedisEndpoint.Port",
  "/properties/RedisEndpoint.Address"
],

Prefixed with /Properties/ (uppercase P)

I assume this is a mistake. Only example is AWS::DMS::ReplicationConfig:

"readOnlyProperties": ["/Properties/ReplicationConfigArn"],

additionalProperties needs to be enforced to false

At several points in the schema, there are JSON schema blocks that force additionalProperties to be false:

https://github.com/aws-cloudformation/aws-cloudformation-resource-schema/blob/6db79a6253e65ee97c55aacfdb7c67f408d2d573/src/main/resources/schema/provider.definition.schema.v1.json#L167-L171

However at no point in the schema is the field marked as required so that this validation applies. This poses a problem since the default value is true.

We need to make a change so this field has to be present. Adding it to the required arrays where necessary is one possible avenue.

Specify properties for serialized behavior

I'd like to create an AWS::S3::BucketNotification resource to fix a longstanding problem in CloudFormation. And when I do, I'd like to enable individual notification configuration items to be managed separately. But because the API only supports updating the entire notification configuration, I need to serialize access to the notification configuration for a given bucket. I plan to do with a resource provider that relies on a user-account-side mechanism involving an SQS FIFO queue. It would be great if, in my schema, I could specify a property or set of properties that would cause all resources with the same values for those properties to have their requests serialized (but resources with different values for those properties could be processed in parallel).

Inconsistent pattern for AWS::Backup::BackupSelection BackupSelection.SelectionName property in eu-central-1

Hi,

I noticed the pattern requirement for the BackupSelection.SelectionName property of the AWS::Backup::BackupSelection resource in eu-central-1 is different to other regions.

The schema definitions for this property are as follows:

eu-central-1

"SelectionName" : {
  "type" : "string",
  "pattern" : "^[a-zA-Z0-9\\-\\_\\.]{1,50}$"
}

us-west-1

"SelectionName" : {
  "type" : "string"
}

ap-southeast-2

"SelectionName" : {
  "type" : "string"
}

Other regions appear to be consistent with us-west-1 and ap-southeast-2, having no pattern requirement.

The impact of this is that we cannot deploy BackupSelection resources in eu-central-1 with the same SelectionName as those in other regions. I think the schema in eu-central-1 must have been changed recently because we still have stacks deployed there which violate the pattern, we're unable to update those stacks without modifying the SelectionName.

Improve error message for pattern mismatch

Currently, schema validation error for pattern mismatch returns error in the form "#/StringProperty: failed validation constraint for keyword [pattern]". It would be good to include pattern value in the error message.

Schematize nuanced immutability (conditional resource replacement, downtime, etc.)

Currently, the CloudFormation resource specification and the documentation do not distinguish the difference between an update operation which fully replaces a resource (including data loss) from those updates which cause some other kind of downtime (e.g a server or database reboot).

The documentation maps both of these concepts to 'Update requires: some interruption' and the resource spec maps only the fully immutable replacement operation, as does the schema currently.

We could include a schema property to signal that a property may cause downtime without data loss as a distinguishing concept.

Maybe support primary identifiers that are list-valued?

Currently, there's an issue where if a primary identifier is list-valued, registration will succeed but execution will fail at runtime with "Internal error" and no logs output. That issue has been reported.

Separately, though: should list-valued identifiers be allowed?

How this came up is we're creating a resource provider for the sorely-needed S3 bucket notification resource. While the notifications have individual IDs, there is a constraint that no two notifications can overlap in prefix+events. So we had this idea of using that combination as an identifier. When we tried it out, it broke as described above.

This case is less of a list-valued identifier, and more of a set-valued constraint, but I wanted to open an issue for discussion if I or others had ideas about when a list-valued identifier would be useful. Given that the primary identifier itself can be list-valued, it seems plausible that sometimes the list wouldn't come from a series of scalar properties but from a single property containing that list.

Add code examples property to schema

It would be great (mostly for documentation generation) if the schema top-level could have an optional array of example model values. Something like:

{
    "typeName": "Foo::Bar::Baz",
    "properties": {
        "someprop": {
            "type": "string"
        }
    },
    ...
    "templateExamples": [
        {
            "description": "A basic example",
            "model": {
                "someprop": "An example value"
            }
        }
    ]
}

Labels: enhancement

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.