aws-samples/ecs-refarch-cloudformation

A reference architecture for deploying containerized microservices with Amazon ECS and AWS CloudFormation (YAML)

Deploying Microservices with Amazon ECS, AWS CloudFormation, and an Application Load Balancer

This reference architecture provides a set of YAML templates for deploying microservices to Amazon EC2 Container Service (Amazon ECS) with AWS CloudFormation.

You can launch this CloudFormation stack in your account:

AWS Region	Short name
US East (Ohio)	us-east-2
US East (N. Virginia)	us-east-1
US GovCloud	us-gov-west-1
US West (Oregon)	us-west-2
US West (N. California)	us-west-1
Canada (Central)	ca-central-1
EU (Paris)	eu-west-3
EU (London)	eu-west-2
EU (Ireland)	eu-west-1
EU (Frankfurt)	eu-central-1
Asia Pacific (Seoul)	ap-northeast-2
Asia Pacific (Tokyo)	ap-northeast-1
Asia Pacific (Sydney)	ap-southeast-2
Asia Pacific (Singapore)	ap-southeast-1
Asia Pacific (Mumbai)	ap-south-1
South America (São Paulo)	sa-east-1

Overview

The repository consists of a set of nested templates that deploy the following:

• A tiered VPC with public and private subnets, spanning an AWS region.
• A highly available ECS cluster deployed across two Availability Zones in an Auto Scaling group and that are AWS SSM enabled.
• A pair of NAT gateways (one in each zone) to handle outbound traffic.
• Two interconnecting microservices deployed as ECS services (website-service and product-service).
• An Application Load Balancer (ALB) to the public subnets to handle inbound traffic.
• ALB path-based routes for each ECS service to route the inbound traffic to the correct service.
• Centralized container logging with Amazon CloudWatch Logs.
• A Lambda Function and Auto Scaling Lifecycle Hook to drain Tasks from your Container Instances when an Instance is selected for Termination in your Auto Scaling Group.

Why use AWS CloudFormation with Amazon ECS?

Using CloudFormation to deploy and manage services with ECS has a number of nice benefits over more traditional methods (AWS CLI, scripting, etc.).

Infrastructure-as-Code

A template can be used repeatedly to create identical copies of the same stack (or to use as a foundation to start a new stack). Templates are simple YAML- or JSON-formatted text files that can be placed under your normal source control mechanisms, stored in private or public locations such as Amazon S3, and exchanged via email. With CloudFormation, you can see exactly which AWS resources make up a stack. You retain full control and have the ability to modify any of the AWS resources created as part of a stack.

Self-documenting

Fed up with outdated documentation on your infrastructure or environments? Still keep manual documentation of IP ranges, security group rules, etc.?

With CloudFormation, your template becomes your documentation. Want to see exactly what you have deployed? Just look at your template. If you keep it in source control, then you can also look back at exactly which changes were made and by whom.

Intelligent updating & rollback

CloudFormation not only handles the initial deployment of your infrastructure and environments, but it can also manage the whole lifecycle, including future updates. During updates, you have fine-grained control and visibility over how changes are applied, using functionality such as change sets, rolling update policies and stack policies.

Template details

The templates below are included in this repository and reference architecture:

Template	Description
master.yaml	This is the master template - deploy it to CloudFormation and it includes all of the others automatically.
infrastructure/vpc.yaml	This template deploys a VPC with a pair of public and private subnets spread across two Availability Zones. It deploys an Internet gateway, with a default route on the public subnets. It deploys a pair of NAT gateways (one in each zone), and default routes for them in the private subnets.
infrastructure/security-groups.yaml	This template contains the security groups required by the entire stack. They are created in a separate nested template, so that they can be referenced by all of the other nested templates.
infrastructure/load-balancers.yaml	This template deploys an ALB to the public subnets, which exposes the various ECS services. It is created in in a separate nested template, so that it can be referenced by all of the other nested templates and so that the various ECS services can register with it.
infrastructure/ecs-cluster.yaml	This template deploys an ECS cluster to the private subnets using an Auto Scaling group and installs the AWS SSM agent with related policy requirements.
infrastructure/lifecyclehook.yaml	This template deploys a Lambda Function and Auto Scaling Lifecycle Hook to drain Tasks from your Container Instances when an Instance is selected for Termination in your Auto Scaling Group.
services/product-service/service.yaml	This is an example of a long-running ECS service that serves a JSON API of products. For the full source for the service, see services/product-service/src.
services/website-service/service.yaml	This is an example of a long-running ECS service that needs to connect to another service (product-service) via the load-balanced URL. We use an environment variable to pass the product-service URL to the containers. For the full source for this service, see services/website-service/src.

After the CloudFormation templates have been deployed, the stack outputs contain a link to the load-balanced URLs for each of the deployed microservices.

The ECS instances should also appear in the Managed Instances section of the EC2 console.

How do I...?

Get started and deploy this into my AWS account

You can launch this CloudFormation stack in your account:

AWS Region	Short name
US East (Ohio)	us-east-2
US East (N. Virginia)	us-east-1
US GovCloud	us-gov-west-1
US West (Oregon)	us-west-2
US West (N. California)	us-west-1
Canada (Central)	ca-central-1
EU (Frankfurt)	eu-west-3
EU (London)	eu-west-2
EU (Ireland)	eu-west-1
EU (Frankfurt)	eu-central-1
Asia Pacific (Seoul)	ap-northeast-2
Asia Pacific (Tokyo)	ap-northeast-1
Asia Pacific (Sydney)	ap-southeast-2
Asia Pacific (Singapore)	ap-southeast-1
Asia Pacific (Mumbai)	ap-south-1
South America (São Paulo)	sa-east-1

Customize the templates

1. Fork this GitHub repository.
2. Clone the forked GitHub repository to your local machine.
3. Modify the templates.
4. Upload them to an Amazon S3 bucket of your choice.
5. Either create a new CloudFormation stack by deploying the master.yaml template, or update your existing stack with your version of the templates.

Create a new ECS service

1. Push your container to a registry somewhere (e.g., Amazon ECR).
2. Copy one of the existing service templates in services/*.
3. Update the ContainerName and Image parameters to point to your container image instead of the example container.
4. Increment the ListenerRule priority number (no two services can have the same priority number - this is used to order the ALB path based routing rules).
5. Copy one of the existing service definitions in master.yaml and point it at your new service template. Specify the HTTP Path at which you want the service exposed.
6. Deploy the templates as a new stack, or as an update to an existing stack.

Setup centralized container logging

By default, the containers in your ECS tasks/services are already configured to send log information to CloudWatch Logs and retain them for 365 days. Within each service's template (in services/*), a LogGroup is created that is named after the CloudFormation stack. All container logs are sent to that CloudWatch Logs log group.

You can view the logs by looking in your CloudWatch Logs console (make sure you are in the correct AWS region).

ECS also supports other logging drivers, including syslog, journald, splunk, gelf, json-file, and fluentd. To configure those instead, adjust the service template to use the alternative LogDriver. You can also adjust the log retention period from the default 365 days by tweaking the RetentionInDays parameter.

For more information, see the LogConfiguration API operation.

Change the ECS host instance type

This is specified in the master.yaml template.

By default, t2.large instances are used, but you can change this by modifying the following section:

ECS:
  Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: ...
      Parameters:
        ... 
        InstanceType: t2.large
        InstanceCount: 4
        ... 

Adjust the Auto Scaling parameters for ECS hosts and services

The Auto Scaling group scaling policy provided by default launches and maintains a cluster of 4 ECS hosts distributed across two Availability Zones (min: 4, max: 4, desired: 4).

It is not set up to scale automatically based on any policies (CPU, network, time of day, etc.).

If you would like to configure policy or time-based automatic scaling, you can add the ScalingPolicy property to the AutoScalingGroup deployed in infrastructure/ecs-cluster.yaml.

As well as configuring Auto Scaling for the ECS hosts (your pool of compute), you can also configure scaling each individual ECS service. This can be useful if you want to run more instances of each container/task depending on the load or time of day (or a custom CloudWatch metric). To do this, you need to create AWS::ApplicationAutoScaling::ScalingPolicy within your service template.

Deploy multiple environments (e.g., dev, test, pre-production)

Deploy another CloudFormation stack from the same set of templates to create a new environment. The stack name provided when deploying the stack is prefixed to all taggable resources (e.g., EC2 instances, VPCs, etc.) so you can distinguish the different environment resources in the AWS Management Console.

Change the VPC or subnet IP ranges

This set of templates deploys the following network design:

Item	CIDR Range	Usable IPs	Description
VPC	10.180.0.0/16	65,536	The whole range used for the VPC and all subnets
Public Subnet	10.180.8.0/21	2,041	The public subnet in the first Availability Zone
Public Subnet	10.180.16.0/21	2,041	The public subnet in the second Availability Zone
Private Subnet	10.180.24.0/21	2,041	The private subnet in the first Availability Zone
Private Subnet	10.180.32.0/21	2,041	The private subnet in the second Availability Zone

You can adjust the CIDR ranges used in this section of the master.yaml template:

VPC:
  Type: AWS::CloudFormation::Stack
    Properties:
      TemplateURL: !Sub ${TemplateLocation}/infrastructure/vpc.yaml
      Parameters:
        EnvironmentName:    !Ref AWS::StackName
        VpcCIDR:            10.180.0.0/16
        PublicSubnet1CIDR:  10.180.8.0/21
        PublicSubnet2CIDR:  10.180.16.0/21
        PrivateSubnet1CIDR: 10.180.24.0/21
        PrivateSubnet2CIDR: 10.180.32.0/21

Update an ECS service to a new Docker image version

ECS has the ability to perform rolling upgrades to your ECS services to minimize downtime during deployments. For more information, see Updating a Service.

To update one of your services to a new version, adjust the Image parameter in the service template (in services/* to point to the new version of your container image. For example, if 1.0.0 was currently deployed and you wanted to update to 1.1.0, you could update it as follows:

TaskDefinition:
  Type: AWS::ECS::TaskDefinition
  Properties:
    ContainerDefinitions:
      - Name: your-container
        Image: registry.example.com/your-container:1.1.0

After you've updated the template, update the deployed CloudFormation stack; CloudFormation and ECS handle the rest.

To adjust the rollout parameters (min/max number of tasks/containers to keep in service at any time), you need to configure DeploymentConfiguration for the ECS service.

For example:

Service: 
  Type: AWS::ECS::Service
    Properties: 
      ...
      DesiredCount: 4
      DeploymentConfiguration: 
        MaximumPercent: 200
        MinimumHealthyPercent: 50

Use the SSM Run Command function to see details in the ECS instances

The AWS SSM Run Command function, in the EC2 console, can be used to execute commands at the shell on the ECS instances. These can be helpful for examining the installed configuration of the instances without requiring direct access to them.

Spot Instances and the Hibernate Agent.

In order to use Spot with this template, you will need to enable SpotPrice under the AWS::AutoScaling::LaunchConfiguration or add in AWS::EC2::SpotFleet support. To fully use Hibernation with Spot instances, please review Spot Instance Interruptions.

Add a new item to this list

If you found yourself wishing this set of frequently asked questions had an answer for a particular problem, please submit a pull request. The chances are that others will also benefit from having the answer listed here.

Contributing

Please create a new GitHub issue for any feature requests, bugs, or documentation improvements.

Where possible, please also submit a pull request for the change.

License

Copyright 2011-2016 Amazon.com, Inc. or its affiliates. All Rights Reserved.

Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at

http://aws.amazon.com/apache2.0/

or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

ecs-refarch-cloudformation's Issues

EC2 not registering to ECS, Container Services are not running

Hi,

Using your template, i have configured launch template with auto scaling group. Reason to use launch template is i want to use mixed instance policy. I have successfully able to create mixed instance based on spot and on demand. The Problem is that i am unable to get EC2 To register to ECS. Could not figure out the reason why. Also another problem i am facing even if i am not using launch template that my container restarts again and again.

For First Problem, Sharing Launch Template and Auto Scaling Group Information which is changed

  ECSAutoScalingGroup:
    DependsOn: ECSCluster
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      MixedInstancesPolicy:
        InstancesDistribution:
          OnDemandAllocationStrategy: prioritized
          OnDemandBaseCapacity: 0
          OnDemandPercentageAboveBaseCapacity: 50
          SpotInstancePools: 20
          SpotAllocationStrategy: lowest-price          
        LaunchTemplate:
          LaunchTemplateSpecification: 
            LaunchTemplateId: !Ref ECSLaunchConfiguration
            Version: 1
          Overrides: 
            - InstanceType: !Select [0, !Split [ ",", !Ref InstanceTypesOverride ] ]
            - InstanceType: !Select [1, !Split [ ",", !Ref InstanceTypesOverride ] ]
      VPCZoneIdentifier: 
        !Ref Subnets
      MinSize: 2
      MaxSize: 5
      DesiredCapacity: 2
      HealthCheckGracePeriod: 90
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName} ECS host
          PropagateAtLaunch: true
    CreationPolicy:
      ResourceSignal:
        Timeout: PT15M
    UpdatePolicy:
      AutoScalingRollingUpdate:
        MinInstancesInService: 1
        MaxBatchSize: 1
        PauseTime: PT15M
        SuspendProcesses:
          - HealthCheck
          - ReplaceUnhealthy
          - AZRebalance
          - AlarmNotification
          - ScheduledActions
        WaitOnResourceSignals: true
  
  ECSLaunchConfiguration:
    Type: AWS::EC2::LaunchTemplate
    Properties:
      LaunchTemplateData:
        ImageId: !Ref ECSAMI
        InstanceType: !Ref InstanceType
        IamInstanceProfile:
          Arn: !GetAtt ECSInstanceProfile.Arn
        UserData:
          "Fn::Base64": !Sub |
            #!/bin/bash
            yum install -y https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm
            yum install -y https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
            yum install -y aws-cfn-bootstrap hibagent 
            /opt/aws/bin/cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource ECSLaunchConfiguration
            /opt/aws/bin/cfn-signal -e 0 --region ${AWS::Region} --stack ${AWS::StackName} --resource ECSAutoScalingGroup
            /usr/bin/enable-ec2-spot-hibernation
    Metadata:
      AWS::CloudFormation::Init:
        config:
          packages:
            yum:
              collectd: []

          commands:
            01_add_instance_to_cluster:
              command: !Sub echo ECS_CLUSTER=ECSCluster >> /etc/ecs/ecs.config
            02_enable_cloudwatch_agent:
              command: !Sub /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:${ECSCloudWatchParameter} -s
          files:
            /etc/cfn/cfn-hup.conf:
              mode: 000400
              owner: root
              group: root
              content: !Sub |
                [main]
                stack=${AWS::StackId}
                region=${AWS::Region}

            /etc/cfn/hooks.d/cfn-auto-reloader.conf:
              content: !Sub |
                [cfn-auto-reloader-hook]
                triggers=post.update
                path=Resources.ECSLaunchConfiguration.Metadata.AWS::CloudFormation::Init
                action=/opt/aws/bin/cfn-init -v --region ${AWS::Region} --stack ${AWS::StackName} --resource ECSLaunchConfiguration

          services:
            sysvinit:
              cfn-hup:
                enabled: true
                ensureRunning: true
                files:
                  - /etc/cfn/cfn-hup.conf
                  - /etc/cfn/hooks.d/cfn-auto-reloader.conf
`
```
The Current Role and IAM Defined
```
  ECSRole:
    Type: AWS::IAM::Role
    Properties:
      Path: /
      RoleName: !Sub ${EnvironmentName}-ECSRole-${AWS::Region}
      AssumeRolePolicyDocument: |
        {
            "Statement": [{
                "Action": "sts:AssumeRole",
                "Effect": "Allow",
                "Principal": {
                    "Service": "ec2.amazonaws.com"
                }
            }]
        }
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM
        - arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
      Policies:
        - PolicyName: ecs-service
          PolicyDocument: |
            {
                "Statement": [{
                    "Effect": "Allow",
                    "Action": [
                        "ecs:CreateCluster",
                        "ecs:DeregisterContainerInstance",
                        "ecs:RegisterContainerInstance", 
                        "ecs:DiscoverPollEndpoint",
                        "ecs:Poll",
                        "ecs:StartTelemetrySession",
                        "ecs:Submit*",
                        "ecr:BatchCheckLayerAvailability",
                        "ecr:BatchGetImage",
                        "ecr:GetDownloadUrlForLayer",
                        "ecr:GetAuthorizationToken",
                        "logs:CreateLogStream",
                        "logs:PutLogEvents"
                    ],
                    "Resource": "*"
                }]
            }

  ECSInstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Path: /
      Roles:
        - !Ref ECSRole

  ECSServiceAutoScalingRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          Action:
            - "sts:AssumeRole"
          Effect: Allow
          Principal:
            Service:
              - application-autoscaling.amazonaws.com
      Path: /
      Policies:
        - PolicyName: ecs-service-autoscaling
          PolicyDocument:
            Statement:
              Effect: Allow
              Action:
                - application-autoscaling:*
                - cloudwatch:DescribeAlarms
                - cloudwatch:PutMetricAlarm
                - ecs:DescribeServices
                - ecs:UpdateService
              Resource: "*"

```
As much as i have researched either it is problem with Role or it is a problem with Internet not working as ec2 instance could not register because of that.

For Second Problem
I have not changed anything. But docker container is showing error like this:

```
2019-06-14T21:30:40Z : task not steady state or terminal; progressing it
2019-06-14T21:30:40Z : waiting for event for task
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:nginxlatest for container website-service concurrently
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: recording timestamp for starting image pulltime: 2019-06-14 21:30:40.071736536 +0000 UTC m=+222.016805805
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:phplatest for container website-service2 concurrently
2019-06-14T21:30:40Z [INFO] Updating container reference website-service2 in Image State - sha256:9467facf170f6e6c7c6cdfbc4f1a6709c21d9cfa172df2fd632a32f639d73b6d
2019-06-14T21:30:40Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: finished pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:phplatest for container website-service2 in 78.076184ms
2019-06-14T21:30:40Z : got container [website-service2] event: [PULLED]
2019-06-14T21:30:40Z : handling container change [{PULLED { <nil> [] <nil> [] map[] 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } <nil>} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:40Z : unable to create state change event for container [website-service2]: create container state change event api: status not recognized by ECS: PULLED
2019-06-14T21:30:40Z : task not steady state or terminal; progressing it
2019-06-14T21:30:40Z : waiting for event for task
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: creating container: website-service2
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: created container name mapping for task: website-service2 -> ecs-website-service-23-website-service2-e4d0f9c0f798e9100000
2019-06-14T21:30:40Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:40Z [INFO] Updating container reference website-service in Image State - sha256:0ba9e8673a64e04d326b2a3e1b00b1da35de9d727a3146effa6d78aa1bd914a7
2019-06-14T21:30:40Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: finished pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:nginxlatest for container website-service in 258.146245ms
2019-06-14T21:30:40Z : got container [website-service] event: [PULLED]
2019-06-14T21:30:40Z : handling container change [{PULLED { <nil> [] <nil> [] map[] 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } <nil>} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:40Z : unable to create state change event for container [website-service]: create container state change event api: status not recognized by ECS: PULLED
2019-06-14T21:30:40Z : task not steady state or terminal; progressing it
2019-06-14T21:30:40Z : waiting for event for task
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: creating container: website-service
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: created container name mapping for task: website-service -> ecs-website-service-23-website-service-de86a6cfdaa89ab87f00
2019-06-14T21:30:40Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: created docker container for task: website-service2 -> a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: created docker container for task: website-service2 -> a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2, took 446.127925ms
2019-06-14T21:30:40Z : got container [website-service2] event: [CREATED]
2019-06-14T21:30:40Z : handling container change [{CREATED {a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2 <nil> [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:40.15925379 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000749300} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:40Z : unable to create state change event for container [website-service2]: create container state change event api: status not recognized by ECS: CREATED
2019-06-14T21:30:40Z : task not steady state or terminal; progressing it
2019-06-14T21:30:40Z : waiting for event for task
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: starting container: website-service2
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: created docker container for task: website-service -> 22f3c9863ef495073ed4b873639a7f9347c79788d489aa2829e762bafea3ce3c
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: created docker container for task: website-service -> 22f3c9863ef495073ed4b873639a7f9347c79788d489aa2829e762bafea3ce3c, took 457.2088ms
2019-06-14T21:30:40Z : got container [website-service] event: [CREATED]
2019-06-14T21:30:40Z : handling container change [{CREATED {22f3c9863ef495073ed4b873639a7f9347c79788d489aa2829e762bafea3ce3c <nil> [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:40.338785118 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000749c00} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:40Z : unable to create state change event for container [website-service]: create container state change event api: status not recognized by ECS: CREATED
2019-06-14T21:30:40Z [INFO] api/task: Updating task's known status to: CREATED, task: website-service:23 arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e, TaskStatus: (NONE->RUNNING) Containers: [website-service2 (CREATED->RUNNING),website-service (CREATED->RUNNING),]
2019-06-14T21:30:40Z : container change also resulted in task change [website-service]: [RUNNING]
2019-06-14T21:30:40Z : unable to create task state change event []: create task state change event api: status not recognized by ECS: CREATED
2019-06-14T21:30:40Z : task not steady state or terminal; progressing it
2019-06-14T21:30:40Z : waiting for event for task
2019-06-14T21:30:40Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: starting container: website-service
2019-06-14T21:30:41Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: started docker container for task: website-service2 -> a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2, took 449.691526ms
2019-06-14T21:30:41Z : got container [website-service2] event: [RUNNING]
2019-06-14T21:30:41Z : handling container change [{RUNNING {a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2 <nil> [{8000 32794 0.0.0.0 0}] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:40.15925379 +0000 UTC 2019-06-14 21:30:41.041611471 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000772800} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:41Z : sending container change event [website-service2]: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service2 -> RUNNING, Ports [{8000 32794 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:41Z : sent container change event [website-service2]: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service2 -> RUNNING, Ports [{8000 32794 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:41Z : task not steady state or terminal; progressing it
2019-06-14T21:30:41Z : waiting for event for task
2019-06-14T21:30:41Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service2 -> RUNNING, Ports [{8000 32794 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:41Z : got container [website-service2] event: [RUNNING]
2019-06-14T21:30:41Z : handling container change [{RUNNING {a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2 <nil> [{8000 32794 0.0.0.0 0}] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:40.15925379 +0000 UTC 2019-06-14 21:30:41.041611471 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000772d00} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:41Z : redundant container state change. website-service2 to RUNNING, but already RUNNING
2019-06-14T21:30:41Z : task not steady state or terminal; progressing it
2019-06-14T21:30:41Z : waiting for event for task
2019-06-14T21:30:41Z : got container [website-service] event: [RUNNING]
2019-06-14T21:30:41Z : handling container change [{RUNNING {22f3c9863ef495073ed4b873639a7f9347c79788d489aa2829e762bafea3ce3c <nil> [{80 32795 0.0.0.0 0}] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:40.338785118 +0000 UTC 2019-06-14 21:30:41.244245241 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000773100} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:41Z : sending container change event [website-service]: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service -> RUNNING, Ports [{80 32795 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:41Z : sent container change event [website-service]: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service -> RUNNING, Ports [{80 32795 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:41Z [INFO] api/task: Updating task's known status to: RUNNING, task: website-service:23 arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e, TaskStatus: (CREATED->RUNNING) Containers: [website-service2 (RUNNING->RUNNING),website-service (RUNNING->RUNNING),]
2019-06-14T21:30:41Z : container change also resulted in task change [website-service]: [RUNNING]
2019-06-14T21:30:41Z : sending task change event [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e -> RUNNING, Known Sent: NONE, PullStartedAt: 2019-06-14 21:30:40.071736536 +0000 UTC m=+222.016805805, PullStoppedAt: 2019-06-14 21:30:40.329906311 +0000 UTC m=+222.274975562, ExecutionStoppedAt: 0001-01-01 00:00:00 +0000 UTC]
2019-06-14T21:30:41Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service -> RUNNING, Ports [{80 32795 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:41Z [INFO] TaskHandler: Adding event: TaskChange: [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e -> RUNNING, Known Sent: NONE, PullStartedAt: 2019-06-14 21:30:40.071736536 +0000 UTC m=+222.016805805, PullStoppedAt: 2019-06-14 21:30:40.329906311 +0000 UTC m=+222.274975562, ExecutionStoppedAt: 0001-01-01 00:00:00 +0000 UTC, arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service2 -> RUNNING, Ports [{8000 32794 0.0.0.0 0}], Known Sent: NONE, arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service -> RUNNING, Ports [{80 32795 0.0.0.0 0}], Known Sent: NONE] sent: false
2019-06-14T21:30:41Z [INFO] TaskHandler: Sending task change: TaskChange: [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e -> RUNNING, Known Sent: NONE, PullStartedAt: 2019-06-14 21:30:40.071736536 +0000 UTC m=+222.016805805, PullStoppedAt: 2019-06-14 21:30:40.329906311 +0000 UTC m=+222.274975562, ExecutionStoppedAt: 0001-01-01 00:00:00 +0000 UTC, arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service2 -> RUNNING, Ports [{8000 32794 0.0.0.0 0}], Known Sent: NONE, arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service -> RUNNING, Ports [{80 32795 0.0.0.0 0}], Known Sent: NONE] sent: false
2019-06-14T21:30:41Z : sent task change event [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e -> RUNNING, Known Sent: NONE, PullStartedAt: 2019-06-14 21:30:40.071736536 +0000 UTC m=+222.016805805, PullStoppedAt: 2019-06-14 21:30:40.329906311 +0000 UTC m=+222.274975562, ExecutionStoppedAt: 0001-01-01 00:00:00 +0000 UTC]
2019-06-14T21:30:41Z : task at steady state: RUNNING
2019-06-14T21:30:41Z : waiting for event for task
2019-06-14T21:30:41Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: started docker container for task: website-service -> 22f3c9863ef495073ed4b873639a7f9347c79788d489aa2829e762bafea3ce3c, took 456.339967ms
2019-06-14T21:30:41Z : got container [website-service] event: [RUNNING]
2019-06-14T21:30:41Z : handling container change [{RUNNING {22f3c9863ef495073ed4b873639a7f9347c79788d489aa2829e762bafea3ce3c <nil> [{80 32795 0.0.0.0 0}] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:40.338785118 +0000 UTC 2019-06-14 21:30:41.244245241 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000773a00} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:41Z : redundant container state change. website-service to RUNNING, but already RUNNING
2019-06-14T21:30:41Z : task at steady state: RUNNING
2019-06-14T21:30:41Z : waiting for event for task
2019-06-14T21:30:41Z : got container [website-service] event: [STOPPED]
2019-06-14T21:30:41Z : handling container change [{STOPPED {22f3c9863ef495073ed4b873639a7f9347c79788d489aa2829e762bafea3ce3c 0xc0007e3be0 [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:40.338785118 +0000 UTC 2019-06-14 21:30:41.244245241 +0000 UTC 2019-06-14 21:30:41.308321597 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000439100} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:41Z [INFO] Task [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: recording execution stopped time. Essential container [website-service] stopped at: 2019-06-14 21:30:41.563972205 +0000 UTC m=+223.509041470
2019-06-14T21:30:41Z : sending container change event [website-service]: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service -> STOPPED, Exit 0, , Ports [{80 32795 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:41Z : sent container change event [website-service]: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service -> STOPPED, Exit 0, , Ports [{80 32795 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:41Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e, TaskStatus: (RUNNING->RUNNING) Containers: [website-service2 (RUNNING->RUNNING),website-service (STOPPED->RUNNING),]]
2019-06-14T21:30:41Z : task not steady state or terminal; progressing it
2019-06-14T21:30:41Z : waiting for event for task
2019-06-14T21:30:41Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e]: stopping container [website-service2]
2019-06-14T21:30:41Z [WARN] DockerGoClient: Unable to decode stats for container 22f3c9863ef495073ed4b873639a7f9347c79788d489aa2829e762bafea3ce3c: context canceled
2019-06-14T21:30:41Z [INFO] Container 22f3c9863ef495073ed4b873639a7f9347c79788d489aa2829e762bafea3ce3c is terminal, stopping stats collection
2019-06-14T21:30:41Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service -> STOPPED, Exit 0, , Ports [{80 32795 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:41Z : got resource [cgroup] event: [REMOVED]
2019-06-14T21:30:41Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service2]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e, TaskStatus: (RUNNING->STOPPED) Containers: [website-service2 (RUNNING->STOPPED),website-service (STOPPED->STOPPED),]]
2019-06-14T21:30:41Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e, TaskStatus: (RUNNING->STOPPED) Containers: [website-service2 (RUNNING->STOPPED),website-service (STOPPED->STOPPED),]]
2019-06-14T21:30:41Z : task not steady state or terminal; progressing it
2019-06-14T21:30:41Z : waiting for event for task
2019-06-14T21:30:41Z : got container [website-service2] event: [STOPPED]
2019-06-14T21:30:41Z : handling container change [{STOPPED {a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2 0xc000433fa0 [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:40.15925379 +0000 UTC 2019-06-14 21:30:41.041611471 +0000 UTC 2019-06-14 21:30:41.62835061 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000748c00} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:41Z : sending container change event [website-service2]: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service2 -> STOPPED, Exit 0, , Ports [{8000 32794 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:41Z : sent container change event [website-service2]: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service2 -> STOPPED, Exit 0, , Ports [{8000 32794 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:41Z [INFO] api/task: Updating task's known status to: STOPPED, task: website-service:23 arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e, TaskStatus: (RUNNING->STOPPED) Containers: [website-service2 (STOPPED->STOPPED),website-service (STOPPED->STOPPED),]
2019-06-14T21:30:41Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service2]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e, TaskStatus: (STOPPED->STOPPED) Containers: [website-service2 (STOPPED->STOPPED),website-service (STOPPED->STOPPED),]]
2019-06-14T21:30:41Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e, TaskStatus: (STOPPED->STOPPED) Containers: [website-service2 (STOPPED->STOPPED),website-service (STOPPED->STOPPED),]]
2019-06-14T21:30:41Z : container change also resulted in task change [website-service2]: [STOPPED]
2019-06-14T21:30:41Z : sending task change event [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2019-06-14 21:30:40.071736536 +0000 UTC m=+222.016805805, PullStoppedAt: 2019-06-14 21:30:40.329906311 +0000 UTC m=+222.274975562, ExecutionStoppedAt: 2019-06-14 21:30:41.563972205 +0000 UTC m=+223.509041470]
2019-06-14T21:30:41Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service2 -> STOPPED, Exit 0, , Ports [{8000 32794 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:41Z [INFO] TaskHandler: Adding event: TaskChange: [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2019-06-14 21:30:40.071736536 +0000 UTC m=+222.016805805, PullStoppedAt: 2019-06-14 21:30:40.329906311 +0000 UTC m=+222.274975562, ExecutionStoppedAt: 2019-06-14 21:30:41.563972205 +0000 UTC m=+223.509041470, arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service -> STOPPED, Exit 0, , Ports [{80 32795 0.0.0.0 0}], Known Sent: RUNNING, arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service2 -> STOPPED, Exit 0, , Ports [{8000 32794 0.0.0.0 0}], Known Sent: RUNNING] sent: false
2019-06-14T21:30:41Z [INFO] TaskHandler: Sending task change: TaskChange: [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2019-06-14 21:30:40.071736536 +0000 UTC m=+222.016805805, PullStoppedAt: 2019-06-14 21:30:40.329906311 +0000 UTC m=+222.274975562, ExecutionStoppedAt: 2019-06-14 21:30:41.563972205 +0000 UTC m=+223.509041470, arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service -> STOPPED, Exit 0, , Ports [{80 32795 0.0.0.0 0}], Known Sent: RUNNING, arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e website-service2 -> STOPPED, Exit 0, , Ports [{8000 32794 0.0.0.0 0}], Known Sent: RUNNING] sent: false
2019-06-14T21:30:41Z [WARN] DockerGoClient: Unable to decode stats for container a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2: context canceled
2019-06-14T21:30:41Z [INFO] Container a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2 is terminal, stopping stats collection
2019-06-14T21:30:41Z : sent task change event [arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2019-06-14 21:30:40.071736536 +0000 UTC m=+222.016805805, PullStoppedAt: 2019-06-14 21:30:40.329906311 +0000 UTC m=+222.274975562, ExecutionStoppedAt: 2019-06-14 21:30:41.563972205 +0000 UTC m=+223.509041470]
2019-06-14T21:30:41Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service2]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e, TaskStatus: (STOPPED->STOPPED) Containers: [website-service2 (STOPPED->STOPPED),website-service (STOPPED->STOPPED),]]
2019-06-14T21:30:41Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e, TaskStatus: (STOPPED->STOPPED) Containers: [website-service2 (STOPPED->STOPPED),website-service (STOPPED->STOPPED),]]
2019-06-14T21:30:41Z : task has reached stopped. Waiting for container cleanup
2019-06-14T21:30:41Z : waiting for event for task
2019-06-14T21:30:41Z : got container [website-service2] event: [STOPPED]
2019-06-14T21:30:41Z : handling container change [{STOPPED {a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2 0xc0000739a0 [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:40.15925379 +0000 UTC 2019-06-14 21:30:41.041611471 +0000 UTC 2019-06-14 21:30:41.62835061 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000749800} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:41Z : redundant container state change. website-service2 to STOPPED, but already STOPPED
2019-06-14T21:30:41Z : waiting for event for task
2019-06-14T21:30:41Z : got container [website-service2] event: [STOPPED]
2019-06-14T21:30:41Z : handling container change [{STOPPED {a2eb01d93436e39e38825b4d2062c5990e55ba30739ab75bb154c39f307d0cb2 0xc0007e32e0 [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/a32c71ec-2751-45eb-a69b-ea857535ee0e com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:40.15925379 +0000 UTC 2019-06-14 21:30:41.041611471 +0000 UTC 2019-06-14 21:30:41.62835061 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000772000} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:41Z : redundant container state change. website-service2 to STOPPED, but already STOPPED
2019-06-14T21:30:41Z : waiting for event for task
2019-06-14T21:30:45Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: unable to create state change event for container [website-service]: create container state change event api: status not recognized by ECS: NONE
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: unable to create state change event for container [website-service2]: create container state change event api: status not recognized by ECS: NONE
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: unable to create task state change event []: create task state change event api: status not recognized by ECS: NONE
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for any previous stops to complete. Sequence number: 16
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: no longer waiting
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: wait over; ready to move towards status: RUNNING
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task not steady state or terminal; progressing it
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:48Z [INFO] Creating cgroup /ecs/d9ca5414-4561-43a1-a70c-cd7eec66f03b
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: transitioned resource [cgroup] to [CREATED]
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got resource [cgroup] event: [CREATED]
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task not steady state or terminal; progressing it
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:phplatest for container website-service2 concurrently
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: recording timestamp for starting image pulltime: 2019-06-14 21:30:48.313621713 +0000 UTC m=+230.258690971
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:nginxlatest for container website-service concurrently
2019-06-14T21:30:48Z [INFO] Updating container reference website-service2 in Image State - sha256:9467facf170f6e6c7c6cdfbc4f1a6709c21d9cfa172df2fd632a32f639d73b6d
2019-06-14T21:30:48Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: finished pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:phplatest for container website-service2 in 139.450168ms
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service2] event: [PULLED]
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{PULLED { <nil> [] <nil> [] map[] 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } <nil>} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: unable to create state change event for container [website-service2]: create container state change event api: status not recognized by ECS: PULLED
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task not steady state or terminal; progressing it
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: creating container: website-service2
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: created container name mapping for task: website-service2 -> ecs-website-service-23-website-service2-dcd9a2aaaad1fad30a00
2019-06-14T21:30:48Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:48Z [INFO] Updating container reference website-service in Image State - sha256:0ba9e8673a64e04d326b2a3e1b00b1da35de9d727a3146effa6d78aa1bd914a7
2019-06-14T21:30:48Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: finished pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:nginxlatest for container website-service in 319.044494ms
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service] event: [PULLED]
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{PULLED { <nil> [] <nil> [] map[] 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } <nil>} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: unable to create state change event for container [website-service]: create container state change event api: status not recognized by ECS: PULLED
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task not steady state or terminal; progressing it
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: creating container: website-service
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: created container name mapping for task: website-service -> ecs-website-service-23-website-service-d2c2b5a6b0f9bdb3df01
2019-06-14T21:30:48Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: created docker container for task: website-service2 -> 4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: created docker container for task: website-service2 -> 4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57, took 442.616021ms
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service2] event: [CREATED]
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{CREATED {4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57 <nil> [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:48.462448527 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000749a00} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: unable to create state change event for container [website-service2]: create container state change event api: status not recognized by ECS: CREATED
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task not steady state or terminal; progressing it
2019-06-14T21:30:48Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:48Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: starting container: website-service2
2019-06-14T21:30:49Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: created docker container for task: website-service -> 085e7b470d6b92d197d8b1805933629f37b0cdf5f35dfed18fe659c633d33ae6
2019-06-14T21:30:49Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: created docker container for task: website-service -> 085e7b470d6b92d197d8b1805933629f37b0cdf5f35dfed18fe659c633d33ae6, took 450.373682ms
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service] event: [CREATED]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{CREATED {085e7b470d6b92d197d8b1805933629f37b0cdf5f35dfed18fe659c633d33ae6 <nil> [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:48.640878172 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000772c00} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: unable to create state change event for container [website-service]: create container state change event api: status not recognized by ECS: CREATED
2019-06-14T21:30:49Z [INFO] api/task: Updating task's known status to: CREATED, task: website-service:23 arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b, TaskStatus: (NONE->RUNNING) Containers: [website-service (CREATED->RUNNING),website-service2 (CREATED->RUNNING),]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: container change also resulted in task change [website-service]: [RUNNING]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: unable to create task state change event []: create task state change event api: status not recognized by ECS: CREATED
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task not steady state or terminal; progressing it
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:49Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: starting container: website-service
2019-06-14T21:30:49Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: started docker container for task: website-service2 -> 4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57, took 478.226904ms
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service2] event: [RUNNING]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{RUNNING {4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57 <nil> [{8000 32796 0.0.0.0 0}] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:48.462448527 +0000 UTC 2019-06-14 21:30:49.374469164 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000438900} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sending container change event [website-service2]: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service2 -> RUNNING, Ports [{8000 32796 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sent container change event [website-service2]: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service2 -> RUNNING, Ports [{8000 32796 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task not steady state or terminal; progressing it
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:49Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service2 -> RUNNING, Ports [{8000 32796 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service2] event: [RUNNING]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{RUNNING {4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57 <nil> [{8000 32796 0.0.0.0 0}] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:48.462448527 +0000 UTC 2019-06-14 21:30:49.374469164 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000439200} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: redundant container state change. website-service2 to RUNNING, but already RUNNING
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task not steady state or terminal; progressing it
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:49Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: started docker container for task: website-service -> 085e7b470d6b92d197d8b1805933629f37b0cdf5f35dfed18fe659c633d33ae6, took 450.299981ms
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service] event: [RUNNING]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{RUNNING {085e7b470d6b92d197d8b1805933629f37b0cdf5f35dfed18fe659c633d33ae6 <nil> [{80 32797 0.0.0.0 0}] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:48.640878172 +0000 UTC 2019-06-14 21:30:49.534901851 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000439c00} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sending container change event [website-service]: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service -> RUNNING, Ports [{80 32797 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sent container change event [website-service]: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service -> RUNNING, Ports [{80 32797 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:49Z [INFO] api/task: Updating task's known status to: RUNNING, task: website-service:23 arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b, TaskStatus: (CREATED->RUNNING) Containers: [website-service (RUNNING->RUNNING),website-service2 (RUNNING->RUNNING),]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: container change also resulted in task change [website-service]: [RUNNING]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sending task change event [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b -> RUNNING, Known Sent: NONE, PullStartedAt: 2019-06-14 21:30:48.313621713 +0000 UTC m=+230.258690971, PullStoppedAt: 2019-06-14 21:30:48.632803714 +0000 UTC m=+230.577872963, ExecutionStoppedAt: 0001-01-01 00:00:00 +0000 UTC]
2019-06-14T21:30:49Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service -> RUNNING, Ports [{80 32797 0.0.0.0 0}], Known Sent: NONE
2019-06-14T21:30:49Z [INFO] TaskHandler: Adding event: TaskChange: [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b -> RUNNING, Known Sent: NONE, PullStartedAt: 2019-06-14 21:30:48.313621713 +0000 UTC m=+230.258690971, PullStoppedAt: 2019-06-14 21:30:48.632803714 +0000 UTC m=+230.577872963, ExecutionStoppedAt: 0001-01-01 00:00:00 +0000 UTC, arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service2 -> RUNNING, Ports [{8000 32796 0.0.0.0 0}], Known Sent: NONE, arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service -> RUNNING, Ports [{80 32797 0.0.0.0 0}], Known Sent: NONE] sent: false
2019-06-14T21:30:49Z [INFO] TaskHandler: Sending task change: TaskChange: [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b -> RUNNING, Known Sent: NONE, PullStartedAt: 2019-06-14 21:30:48.313621713 +0000 UTC m=+230.258690971, PullStoppedAt: 2019-06-14 21:30:48.632803714 +0000 UTC m=+230.577872963, ExecutionStoppedAt: 0001-01-01 00:00:00 +0000 UTC, arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service2 -> RUNNING, Ports [{8000 32796 0.0.0.0 0}], Known Sent: NONE, arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service -> RUNNING, Ports [{80 32797 0.0.0.0 0}], Known Sent: NONE] sent: false
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sent task change event [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b -> RUNNING, Known Sent: NONE, PullStartedAt: 2019-06-14 21:30:48.313621713 +0000 UTC m=+230.258690971, PullStoppedAt: 2019-06-14 21:30:48.632803714 +0000 UTC m=+230.577872963, ExecutionStoppedAt: 0001-01-01 00:00:00 +0000 UTC]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task at steady state: RUNNING
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service] event: [RUNNING]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{RUNNING {085e7b470d6b92d197d8b1805933629f37b0cdf5f35dfed18fe659c633d33ae6 <nil> [{80 32797 0.0.0.0 0}] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:48.640878172 +0000 UTC 2019-06-14 21:30:49.534901851 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000749300} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: redundant container state change. website-service to RUNNING, but already RUNNING
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task at steady state: RUNNING
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service] event: [STOPPED]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{STOPPED {085e7b470d6b92d197d8b1805933629f37b0cdf5f35dfed18fe659c633d33ae6 0xc000418800 [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:48.640878172 +0000 UTC 2019-06-14 21:30:49.534901851 +0000 UTC 2019-06-14 21:30:49.600689543 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000772500} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:49Z [INFO] Task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: recording execution stopped time. Essential container [website-service] stopped at: 2019-06-14 21:30:49.866531816 +0000 UTC m=+231.811601072
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sending container change event [website-service]: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service -> STOPPED, Exit 0, , Ports [{80 32797 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sent container change event [website-service]: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service -> STOPPED, Exit 0, , Ports [{80 32797 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:49Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b, TaskStatus: (RUNNING->RUNNING) Containers: [website-service (STOPPED->RUNNING),website-service2 (RUNNING->RUNNING),]]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task not steady state or terminal; progressing it
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:49Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: stopping container [website-service2]
2019-06-14T21:30:49Z [WARN] DockerGoClient: Unable to decode stats for container 085e7b470d6b92d197d8b1805933629f37b0cdf5f35dfed18fe659c633d33ae6: context canceled
2019-06-14T21:30:49Z [INFO] Container 085e7b470d6b92d197d8b1805933629f37b0cdf5f35dfed18fe659c633d33ae6 is terminal, stopping stats collection
2019-06-14T21:30:49Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service -> STOPPED, Exit 0, , Ports [{80 32797 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got resource [cgroup] event: [REMOVED]
2019-06-14T21:30:49Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b, TaskStatus: (RUNNING->STOPPED) Containers: [website-service (STOPPED->STOPPED),website-service2 (RUNNING->STOPPED),]]
2019-06-14T21:30:49Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service2]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b, TaskStatus: (RUNNING->STOPPED) Containers: [website-service (STOPPED->STOPPED),website-service2 (RUNNING->STOPPED),]]
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task not steady state or terminal; progressing it
2019-06-14T21:30:49Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service2] event: [STOPPED]
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{STOPPED {4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57 0xc000816500 [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:48.462448527 +0000 UTC 2019-06-14 21:30:49.374469164 +0000 UTC 2019-06-14 21:30:49.92422336 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000773200} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sending container change event [website-service2]: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service2 -> STOPPED, Exit 0, , Ports [{8000 32796 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sent container change event [website-service2]: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service2 -> STOPPED, Exit 0, , Ports [{8000 32796 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:50Z [INFO] api/task: Updating task's known status to: STOPPED, task: website-service:23 arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b, TaskStatus: (RUNNING->STOPPED) Containers: [website-service (STOPPED->STOPPED),website-service2 (STOPPED->STOPPED),]
2019-06-14T21:30:50Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b, TaskStatus: (STOPPED->STOPPED) Containers: [website-service (STOPPED->STOPPED),website-service2 (STOPPED->STOPPED),]]
2019-06-14T21:30:50Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service2]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b, TaskStatus: (STOPPED->STOPPED) Containers: [website-service (STOPPED->STOPPED),website-service2 (STOPPED->STOPPED),]]
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: container change also resulted in task change [website-service2]: [STOPPED]
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sending task change event [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2019-06-14 21:30:48.313621713 +0000 UTC m=+230.258690971, PullStoppedAt: 2019-06-14 21:30:48.632803714 +0000 UTC m=+230.577872963, ExecutionStoppedAt: 2019-06-14 21:30:49.866531816 +0000 UTC m=+231.811601072]
2019-06-14T21:30:50Z [INFO] TaskHandler: batching container event: arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service2 -> STOPPED, Exit 0, , Ports [{8000 32796 0.0.0.0 0}], Known Sent: RUNNING
2019-06-14T21:30:50Z [INFO] TaskHandler: Adding event: TaskChange: [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2019-06-14 21:30:48.313621713 +0000 UTC m=+230.258690971, PullStoppedAt: 2019-06-14 21:30:48.632803714 +0000 UTC m=+230.577872963, ExecutionStoppedAt: 2019-06-14 21:30:49.866531816 +0000 UTC m=+231.811601072, arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service -> STOPPED, Exit 0, , Ports [{80 32797 0.0.0.0 0}], Known Sent: RUNNING, arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service2 -> STOPPED, Exit 0, , Ports [{8000 32796 0.0.0.0 0}], Known Sent: RUNNING] sent: false
2019-06-14T21:30:50Z [INFO] TaskHandler: Sending task change: TaskChange: [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2019-06-14 21:30:48.313621713 +0000 UTC m=+230.258690971, PullStoppedAt: 2019-06-14 21:30:48.632803714 +0000 UTC m=+230.577872963, ExecutionStoppedAt: 2019-06-14 21:30:49.866531816 +0000 UTC m=+231.811601072, arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service -> STOPPED, Exit 0, , Ports [{80 32797 0.0.0.0 0}], Known Sent: RUNNING, arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b website-service2 -> STOPPED, Exit 0, , Ports [{8000 32796 0.0.0.0 0}], Known Sent: RUNNING] sent: false
2019-06-14T21:30:50Z [WARN] DockerGoClient: Unable to decode stats for container 4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57: context canceled
2019-06-14T21:30:50Z [INFO] Container 4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57 is terminal, stopping stats collection
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: sent task change event [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b -> STOPPED, Known Sent: RUNNING, PullStartedAt: 2019-06-14 21:30:48.313621713 +0000 UTC m=+230.258690971, PullStoppedAt: 2019-06-14 21:30:48.632803714 +0000 UTC m=+230.577872963, ExecutionStoppedAt: 2019-06-14 21:30:49.866531816 +0000 UTC m=+231.811601072]
2019-06-14T21:30:50Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b, TaskStatus: (STOPPED->STOPPED) Containers: [website-service (STOPPED->STOPPED),website-service2 (STOPPED->STOPPED),]]
2019-06-14T21:30:50Z [INFO] api/task: Updating task desired status to stopped because of container: [website-service2]; task: [website-service:23 arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b, TaskStatus: (STOPPED->STOPPED) Containers: [website-service (STOPPED->STOPPED),website-service2 (STOPPED->STOPPED),]]
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: task has reached stopped. Waiting for container cleanup
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service2] event: [STOPPED]
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{STOPPED {4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57 0xc0007a0500 [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:48.462448527 +0000 UTC 2019-06-14 21:30:49.374469164 +0000 UTC 2019-06-14 21:30:49.92422336 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000749b00} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: redundant container state change. website-service2 to STOPPED, but already STOPPED
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: got container [website-service2] event: [STOPPED]
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: handling container change [{STOPPED {4b3b2afc91676b3ea7ea534584df4130897172199061fb0f3c5efe7339551b57 0xc000817160 [] <nil> [] map[com.amazonaws.ecs.cluster:testservice1 com.amazonaws.ecs.container-name:website-service2 com.amazonaws.ecs.task-arn:arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b com.amazonaws.ecs.task-definition-family:website-service com.amazonaws.ecs.task-definition-version:23] 2019-06-14 21:30:48.462448527 +0000 UTC 2019-06-14 21:30:49.374469164 +0000 UTC 2019-06-14 21:30:49.92422336 +0000 UTC {UNKNOWN <nil> 0 } default 0xc000773a00} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: redundant container state change. website-service2 to STOPPED, but already STOPPED
2019-06-14T21:30:50Z [INFO] Managed task [arn:aws:ecs:us-east-1:866864877893:task/d9ca5414-4561-43a1-a70c-cd7eec66f03b]: waiting for event for task
2019-06-14T21:30:55Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:58Z  unable to create state change event for container [website-service2]: create container state change event api: status not recognized by ECS: NONE
2019-06-14T21:30:58Z  unable to create state change event for container [website-service]: create container state change event api: status not recognized by ECS: NONE
2019-06-14T21:30:58Z  unable to create task state change event []: create task state change event api: status not recognized by ECS: NONE
2019-06-14T21:30:58Z  waiting for any previous stops to complete. Sequence number: 17
2019-06-14T21:30:58Z  waiting for event for task
2019-06-14T21:30:58Z  no longer waiting
2019-06-14T21:30:58Z  wait over; ready to move towards status: RUNNING
2019-06-14T21:30:58Z  task not steady state or terminal; progressing it
2019-06-14T21:30:58Z  waiting for event for task
2019-06-14T21:30:58Z [INFO] Creating cgroup /ecs/bec63e70-97aa-4e22-9088-3b51980e5dc9
2019-06-14T21:30:58Z  transitioned resource [cgroup] to [CREATED]
2019-06-14T21:30:58Z  got resource [cgroup] event: [CREATED]
2019-06-14T21:30:58Z  task not steady state or terminal; progressing it
2019-06-14T21:30:58Z  waiting for event for task
2019-06-14T21:30:58Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:nginxlatest for container website-service concurrently
2019-06-14T21:30:58Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: recording timestamp for starting image pulltime: 2019-06-14 21:30:58.631339717 +0000 UTC m=+240.576408975
2019-06-14T21:30:58Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:phplatest for container website-service2 concurrently
2019-06-14T21:30:58Z [INFO] Updating container reference website-service2 in Image State - sha256:9467facf170f6e6c7c6cdfbc4f1a6709c21d9cfa172df2fd632a32f639d73b6d
2019-06-14T21:30:58Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:58Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: finished pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:phplatest for container website-service2 in 68.843048ms
2019-06-14T21:30:58Z  got container [website-service2] event: [PULLED]
2019-06-14T21:30:58Z  handling container change [{PULLED { <nil> [] <nil> [] map[] 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } <nil>} ContainerStatusChangeEvent}] for container [website-service2]
2019-06-14T21:30:58Z  unable to create state change event for container [website-service2]: create container state change event api: status not recognized by ECS: PULLED
2019-06-14T21:30:58Z  task not steady state or terminal; progressing it
2019-06-14T21:30:58Z  waiting for event for task
2019-06-14T21:30:58Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: creating container: website-service2
2019-06-14T21:30:58Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: created container name mapping for task: website-service2 -> ecs-website-service-23-website-service2-fac2d6d098faecd52600
2019-06-14T21:30:58Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:58Z [INFO] Updating container reference website-service in Image State - sha256:0ba9e8673a64e04d326b2a3e1b00b1da35de9d727a3146effa6d78aa1bd914a7
2019-06-14T21:30:58Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:58Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: finished pulling image 866864877893.dkr.ecr.us-east-1.amazonaws.com/wpmu1:nginxlatest for container website-service in 229.778037ms
2019-06-14T21:30:58Z  got container [website-service] event: [PULLED]
2019-06-14T21:30:58Z  handling container change [{PULLED { <nil> [] <nil> [] map[] 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC 0001-01-01 00:00:00 +0000 UTC {UNKNOWN <nil> 0 } <nil>} ContainerStatusChangeEvent}] for container [website-service]
2019-06-14T21:30:58Z  unable to create state change event for container [website-service]: create container state change event api: status not recognized by ECS: PULLED
2019-06-14T21:30:58Z  task not steady state or terminal; progressing it
2019-06-14T21:30:58Z  waiting for event for task
2019-06-14T21:30:58Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: creating container: website-service
2019-06-14T21:30:58Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: created container name mapping for task: website-service -> ecs-website-service-23-website-service-fc8e8cdaa685c3f98201
2019-06-14T21:30:58Z [INFO] Saving state! module="statemanager"
2019-06-14T21:30:59Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: created docker container for task: website-service2 -> b4d1ed19ce0c52b11b27c997aa1dc0accc7919d276f2611152d19e1dbfd3232d
2019-06-14T21:30:59Z [INFO] Task engine [arn:aws:ecs:us-east-1:866864877893:task/bec63e70-97aa-4e22-9088-3b51980e5dc9]: created docker container for task: website-service2 -> b4d1ed19ce0c52b11b27c997aa1dc0accc7919d276f2611152d19e1dbfd3232d, took 329.544187ms
2019-06-14T21:30:59Z  got container [website-service2] event: [CREATED]


```

Please guide

Deploying in a private subnet

I've tried deploying these templates into a pre-existing VPC with private subnets (which can route to a NAT gateway).

• security-groups.yaml
• load-balancers.yaml
• ecs-cluster.yaml

The instances don't register to the cluster (they appear in the default cluster in the ECS control panel) and the stack reverts after 15 minutes.

Has anyone tried using these templates like this?

EC2 instance registered on ECS default cluster

I don't know how to reproduce this but a newly created ec2 instance registered itself on ECS default cluster instead of the right one.
Why don't you set ECS_CLUSTER in ecs.config in UserData?

CloudFormation validate-template error on website-service/service.yaml

Before copying CF templates to S3 I typically run validate-template on them using the aws cli. I notice that this error is now appearing when I validate:

aws cloudformation validate-template --template-body file://services/website-service/service.yaml

An error occurred (ValidationError) when calling the ValidateTemplate operation: Template error: if specifying one argument to Fn::GetAtt, that argument must be a non-empty string in format <LogicalId>.<Attribute>

I think that it might be due to the following line (error does not occur if commented out):

RoleARN: !GetAtt ECSServiceAutoScalingRoleARN

Is this expected?

Include Route53 HealthChecks

Suggest adding automatic configuration of Route53 HealthChecks for both website and product service.

I am experimenting with this now but having a problem. The healthchecks work fine with an open ALB but if I change the ALB SG to restrict access and add the HC IPs to the SG the product service HC works but the website service HC does not.
Not sure why this is not working yet but I do think having healthchecks are good to act as examples of monitoring the deployed services.

Will update this as I learn more.

Stack getting rolled back automatically

Deployed the CloudFormation stack as is, in the us-east-1 region, the process got rolled back automatically after some time.

Image for reference:

[PS: I have not initiated any rollbacks]

ECS is not created

I ran your cloudformation templates. The stack is created successfully, but no ECS is created. There is no service, no task definition created. But there is no error. Seems your ecs-cluster yaml is totally ignored.

Cluster Auto Scaling

ECS Cluster Auto Scaling is now GA. It would make sense to add that once support is added in CloudFormation.

Cluster Auto Scaling may also remove the need for a dedicated container instance draining lifecycle hook.

VPC template does not support DNS hostnames

Hi, it's quite possible that this was deliberate, but I notice that your AWS::EC2::VPC resource does not indicate EnableDnsHostnames: true and hence instances in the VPC do not get hostnames.

Update ECS AMI amzn-ami-2018.03.h-amazon-ecs-optimized

amzn-ami-2018.03.h-amazon-ecs-optimized

Nov 2018

• ECS agent: 1.21.0
• Docker: 18.06.1-ce
• ecs-init: 1.21.0-1

Simple question on ALB and NAT gateway traffic

In the document, I saw these:
•A pair of NAT gateways (one in each zone) to handle outbound traffic.
•An Application Load Balancer (ALB) to the public subnets to handle inbound traffic.
I believe these two statements are independent. Meaning:

• internet access request will go out through NAT gateway
• external request from internet to ESC services will come through ALB, and response get sent back through ALB as well.
  It does not mean that external request will be returned through NAT gateway. Is this correct?

make fails for both product-service and website-service

It looks like the gin dependency is not being pulled into the containers at make time:

make
Building Linux binary ready for containerisation...
docker run --rm -it -v "":/gopath -v "/User/Mark/github/aws-samples/ecs-refarch-cloudformation/services/product-service/src":/app -e "GOPATH=/gopath" -w /app golang:1.7 sh -c 'CGO_ENABLED=0 go build -a --installsuffix cgo --ldflags="-s" -o app'
main.go:3:8: cannot find package "github.com/gin-gonic/gin" in any of:
        /usr/local/go/src/github.com/gin-gonic/gin (from $GOROOT)
        /gopath/src/github.com/gin-gonic/gin (from $GOPATH)
make: *** [binary] Error 1

I have the following paths shared in Docker:

/User/Mark/github/aws-samples/ecs-refarch-cloudformation/services/product-service/src
/User/Mark/github/aws-samples/ecs-refarch-cloudformation/services/website-service/src

Running Docker Version 18.03.1-ce-mac65 (24312) on macOS High Sierra

unable to login to Ec2

May i know how i can map my existing keypair to enable logging to Ec2

Implement connection draining

Implement automatic connection draining on ECS hosts, via Auto Scaling Group lifecycle events.

https://aws.amazon.com/blogs/compute/how-to-automate-container-instance-draining-in-amazon-ecs/

Pull requests welcome, otherwise I will try and look at implementing this over the coming weeks.

Integrating CodeBuild/CodePipeline

Potential of including refarch example of CodeBuild/CodePipeline for creating/deploying new Docker images in the stack.

Update AMI IDs

AMI IDs are out of date, creating issue to relate to incoming PR.

Specify a Bucket Name

Add the following parameter in the master.yaml.

Parameters:
  BucketName:
    Type: String
    Description: Name of bucket containing YAML scripts needed to build cluster

And replace the TemplateURL

TemplateURL: !Sub https://s3.amazonaws.com/${BucketName}/infrastructure/vpc.yaml

Not Support China two regions

region:
cn-north-1
cn-northwest-1

IPv6?

It would be great if this could handle IPv6/dualstack traffic

How to create a HTTPS listener

Is there any example of attaching HTTPS listener to the load balancer?

Continuous delivery?

Images are hardcoded in the example.

In a real world scenario, they will be changed over the time.

But because of https://forums.aws.amazon.com/thread.jspa?messageID=741932#741932 it's impossible to define the stack without the hardcoded images.

Could you please update an example to demonstrate how to do CD? Otherwise, it's a bit useless :)

Thanks!

Implicit casting of string to List?

The master template pulls a string output from vpc.yaml into ec-cluster.yaml as a List. Is this possible? Does CloudFormation support implicit casting?

vpc.yaml outputs a string:

    PrivateSubnets:
        Description: A list of the private subnets
        Value: !Join [ ",", [ !Ref PrivateSubnet1, !Ref PrivateSubnet2 ]]

The docs for Join say:

Returns 
The combined string.

ecs-cluster.yaml accepts a list, not a string:

    Subnets:
        Description: Choose which subnets this ECS cluster should be deployed to
        Type: List<AWS::EC2::Subnet::Id>

master.yaml sends the string to ec-cluster.yaml:

                Subnets: !GetAtt VPC.Outputs.PrivateSubnets

How is this possible? CloudFormation AFAIK does not support casting of data types.

ecr access

Trying to pull repository 275396840892.dkr.ecr.us-east-1.amazonaws.com/ecs-refarch-cloudformation/product-service ...
Pulling repository 275396840892.dkr.ecr.us-east-1.amazonaws.com/ecs-refarch-cloudformation/product-service
unauthorized: authentication required

Looks the ecr repository mentioned in service templates is not anonymous access, we beginners can't pull the image in our testing. Can we make the 2 images publicly access? This will greatly help beginners to understand the whole deployment process.

Thanks a lot again for the great sample templates.

Automate updates to ECS Optimized AMIs

Currently this is a manual process, and normally kind members of the community will submit a pull request updating the AMI identifiers in the infrastructure/ecs-cluster.yaml file.

We should either:

1. Use AWS CodeBuild to check for the latest AMIs for each region, and automatically open a pull request for the changes. I've done similar in another project (awslabs/goformation) before and it's worked well.

2. Replace the hardcoded AMIs in the CloudFormation templates with a custom resource that looks up the latest AMI for the region.

I think option 1 would be preferable, as i'm not sure CloudFormation update-stack evaluates the result of custom resources, so would not update stacks properly when the AMIs change.

Wrong permission bits on cfn-init-created files

The permission bits must be quoted. The problem is here:

ecs-refarch-cloudformation/infrastructure/ecs-cluster.yaml

Line 142 in ecacd7a

mode: 000400

When YAML parses that, it produces in terms of JSON:

{"mode": 256}

CloudFormation then turns that into the string '256'. Later, cfn-init parses that using, in Python, int(file['mode'], 8). cfn-init then happily sets the mode of that file to 256: writable by user=root, read-executable by group=root, read-writable by other. This problem became very clear to me when I tried to set the mode to 0444 (292), which isn't a valid octal.

Setting ALB SG to an IP breaks website service (but not the products service)

Setting LoadBalancerSecurityGroup CidrIp in security-groups.yaml to a specific address/32 appears to break the website service. The product service continues to work correctly.
I have observed this behavior both when the stack is created with that set and when leaving it open and then setting it in the ALB SG after the stack is created.

Steps to reproduce:

• set the desired ip value in security-groups.yaml
• create the stack
• get the stack outputs for ProductServiceUrl and WebsiteServiceUrl and attempt to access them

The WebsiteServiceUrl returns a 503 and the ECS Console shows the WebsiteService as cycling between 0 and 2 running tasks.

Is anyone else seeing this behavior?

I have been trying to determine the root cause but no luck so far. I am not sure what is different in the ALB rules to ECS access when a task has a path of / vs /products and the SG is not open to all.

Would like to see a Fargate version of this

I've been working through the process of building out an ECS Fargate implementation of our application. I'm somewhat new to AWS and ECS in particular, so this template has been a godsend. I'd love to see an adaptation for Fargate.

Migrate to Amazon Linux 2 AMI

The use of the recommended ECS-Optimized AMIs is great, let's migrate it though to use the Amazon Linux 2 ones.

Update ecs agent at instance launch?

I was wondering if updating the ecs-agent at ec2 instance launch was a good idea?
I was thinking of just including the following in the ecs-cluster.yaml user data but not sure if there is a reason not to do it:
yum update -y ecs-init

How to update container instances in production?

What's the best way to update the container instances--new size, new AMI? (Thanks for the email notifications about the new ECS optimized AMIs by the way.) Seems like we could update the launch configuration AMI, but I don't want to interrupt existing instances. Would I need to set a DeletionPolicy on the EC2s, update the stack to the new instances, and then go in and manually delete the old ones?

VPC deploy fails when deploying 2nd stack?

I am using a lightly modified version of this stack in my company, and our staging environment has been using it with no trouble. Today I tried to spin up another instance of this stack, but it fails in the VPC step with the following error:

Embedded stack arn:aws:cloudformation:us-east-1:REDACTED:stack/my-environment-VPC-SOME_ID/42f61810-7b94-11e7-812e-50d5ca6e601e was not successfully created: The following resource(s) failed to create: [PrivateSubnet2RouteTableAssociation, PrivateSubnet1RouteTableAssociation, NatGateway1EIP, PublicSubnet1RouteTableAssociation, NatGateway2EIP, DefaultPublicRoute, PublicSubnet2RouteTableAssociation].

The stack name is different than the existing stack from this template.

Do I need to change the CIDR ranges in the master template if it's a new VPC?

Built-in abstractions in CF for ECS

Hi!

Right now we have to copy-paste ECSLaunchConfiguration's magic from stack to stack.

Having to call cf-init sounds like a non-native experience as well:
https://github.com/awslabs/ecs-refarch-cloudformation/blob/master/infrastructure/ecs-cluster.yaml#L107

I'm wondering if CF might provide some built-in abstractions for it? Currently it's looks almost like a hack :(

"Private Subnets + NAT Gateways" versus "Security Groups"

Hello there,

First of all I would like to express a big thank you to you guys for making such a beautiful CF template example for us all to reference on. This example has explained to me how I pass resources beautifully without spaghetti links and loopback issues.

If you don't mind I would like to ask a quick question related to the choice of approach for VPC on the CF template.

---

Problem and Goal

I understand that for security issues, our goal here is, ECS instances should not be accessible from the Internet, but ECS instances needs to be able to "connect out" if in any case this is necessary (downloading packages etc.)

Approach A - by ecs-refarch-cloudformation

I noticed (at least I think) that in the template you guys have made use of private subnets and NAT gateways to achieve this goal.

Approach B - by me (bs-thomas)

To be honest, before I saw this CF template, I had built a cheap amateur version of a similar stack (with much less features of course) and have been using it for some time.

The approach I used to achieve the above goal is to simply declare some security groups for these ECS instances to block the traffic out from the Internet. (On a side note, I have also added a "backdoor" for my company's IP address to be able to access the underlying instances in case we have to do something about them, which is prety convenient)

This achieves a "similar" effect (probably not same), but cheaper approach as I do not need to employ NAT gateways.

My Question

I assume there is a difference, but I don't know what that is and wanted to ask experts like you guys out there.

So my question is, I want to know the pros and cons for the (a) and (b) approaches.

If you guys can shed me some light I'd be highly appreciated!

Assigning multiple TargetGroupArn to a listener

Hi,

I want to run multiple services on ALB using path based routing and I want to assign multiple TargetGroupArn to the listener (as highlighted below), please suggest the right way or provide any CF script to launch multiple services on single ALB.

Thanks

ALBListener:
Type: AWS::ElasticLoadBalancingV2::Listener
DependsOn: ECSServiceRole
Properties:
DefaultActions:
- Type: forward
TargetGroupArn: !Ref 'ECSTG', !Ref 'ECSTest2TG']
LoadBalancerArn: !Ref 'ECSALB'
Port: '80'
Protocol: HTTP

Support ecs-cli

There should be a way to configure ecs-cli in such a way that this stack can be managed via ecs-cli if the need arises. A command such as follows would be sufficient to configure ecs-cli:

ecs-cli configure --region us-west-2 --profile sandbox --cluster ${STACK_NAME} --compose-project-name-prefix "${STACK_NAME}-" --compose-service-name-prefix "${STACK_NAME}-" --cfn-stack-name-prefix "${STACK_NAME}-"

Adding Cost Estimate for this architecture.

Hi, I wanted to add a cost estimate for this architecture. This will give anyone deploying this architecture a rough idea on what the cost will be. I've added a PR. Let me know what you guys think about this idea.

Support Service autoscaling

It is very hard to find any examples that show cloudformation that include service autoscaling
Please shpw us how to do it properly

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html

http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service_autoscaling_tutorial.html

EC2 instance is going to terminate while stack is creating

Hello,

I'm trying to create a stack and faced with an issue.
EC2 container instance is terminating when I create the stack
I have grubbed system log

Starting hibagent...                              Ok
ci-info: no authorized ssh keys fingerprints found for user ec2-user.
ci-info: no authorized ssh keys fingerprints found for user ec2-user.
ec2: 
ec2: #############################################################
ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
ec2: 1024 SHA256:0HRM62KQcPYt7uT5rEfEfdCLTMDwrJg7FkSxvSmI6yc root@ip-172-16-31-15 (DSA)
ec2: 256 SHA256:vmMuN5aJ+N0VDsCG7PxZYm58r4Km0/7ffbFO7QsUtXY root@ip-172-16-31-15 (ECDSA)
ec2: 256 SHA256:o3bALTEo5/kwi+etXHWULhZIQwQPS2v87z5bBRcgtu4 no comment (ED25519)
ec2: 2048 SHA256:DM1q6yvpvgbbwh0ovBMuRsiCVL+Qbg8oy78zRsIM3eg root@ip-172-16-31-15 (RSA)
ec2: -----END SSH HOST KEY FINGERPRINTS-----
ec2: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
ecdsa-sha2-nistp256 *********************T root@ip-172-16-31-15
-----END SSH HOST KEY KEYS-----
Cloud-init v. 0.7.6 finished at Mon, 23 Apr 2018 15:57:37 +0000. Datasource DataSourceEc2.  Up 166.09 seconds
[  166.298678] Adding 4095996k swap on /swap.  Priority:-1 extents:9 across:4644860k SSFS

Amazon Linux AMI release 2017.09
Kernel 4.9.81-35.56.amzn1.x86_64 on an x86_64

ip-172-16-31-15 login: [  172.300108] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
[  172.492575] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
[  172.772403] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
[  173.112984] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
[  173.312292] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
[  173.613613] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
[  173.837826] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
[  173.985410] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
[  174.095265] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
[  174.170559] cgroup: docker-runc (7847) created nested cgroup for controller "memory" which has incomplete hierarchy support. Nested cgroups may change behavior in the future.
[  174.180947] cgroup: "memory" requires setting use_hierarchy to 1 on the root
[  174.534063] EXT4-fs (dm-4): mounted filesystem with ordered data mode. Opts: (null)
/dev/fd/10: line 1: /sbin/plymouthd: No such file or directory
initctl: Event failed
Stopping docker: [  OK  ]
stopping awslogs
Stopping awslogs: [  OK  ]
Stopping sshd:  [  OK  ]
Shutting down sm-client: [  OK  ]
Shutting down sendmail: [  OK  ]
Stopping crond: [  OK  ]
Stopping acpi daemon: [  OK  ]
Shutting down ntpd: [  OK  ]
Stopping block device availability: Deactivating block devices:
  [LVM]: deactivating Volume Group docker... done
[  OK  ]
Stopping auditd: [  269.178746] audit: type=1305 audit(1524499161.030:87): audit_pid=0 old=2276 auid=4294967295 ses=4294967295 res=1
[  OK  ]
[  269.276278] audit: type=1305 audit(1524499161.130:88): audit_enabled=0 old=1 auid=4294967295 ses=4294967295 res=1
Shutting down system logger: [  OK  ]
Shutting down interface eth0:  [  OK  ]
Shutting down loopback interface:  [  OK  ]
Stopping cgconfig service: [  OK  ]
Stopping monitoring for VG docker: [  OK  ]
Stopping rngd: [  OK  ]
Stopping cfn-hup: [  OK  ]
Sending all processes the TERM signal... init: ecs post-stop process (8019) killed by TERM signal
[  OK  ]
Sending all processes the KILL signal... [  OK  ]
Saving random seed:  [  OK  ]
Turning off swap:  [  OK  ]
Unmounting file systems:  [  OK  ]
init: Re-executing /sbin/init
[  276.343996] EXT4-fs (xvda1): re-mounted. Opts: (null)
Halting system...
[  276.494089] ACPI: Preparing to enter system sleep state S5
[  276.499984] reboot: Power down
[  276.507425] acpi_power_off called

Could you point me where to dig further?

Support persistence data

Hi,
Anyone can give an example about Database persistence service.

Thanks a lot

Provide access to ECS instances

When using this template it can be helpful to provide access to the ECS instances to see how they work. This helps getting an understanding of how ECS and runnings tasks work.

Update ECS AMI, 2018-08-14, amzn-ami-2018.03.e

The current Amazon ECS-optimized AMI (amzn-ami-2018.03.e-amazon-ecs-optimized) consists of:

• The latest minimal version of the Amazon Linux AMI
• The latest version of the Amazon ECS container agent (1.20.1)
• The recommended version of Docker for the latest Amazon ECS container agent (18.03.1-ce)
• The latest version of the ecs-init package to run and monitor the Amazon ECS agent (1.20.1-1)

Default to latest ECS-Optimized AMI

Default to latest ECS-Optimized AMI

Added ECSAMI parameter which defaults to latest ECS-Optimized AMI
Removed legacy AMI Mappings
Pull Request #88

Thanks @mrichman for the pointer :)

Restricting access to ALB prevents the website service from reaching the product service

The ALB portion of security-groups.yaml mentions that access to the ALB can be restricted by changing the CIDR, wide open by default. However, if that is changed it breaks the website service from accessing the product service because the request comes from the ECS hosts via the NAT Gateways which will not be allowed.
While the NAT Gateway addresses can be subsequently manually added to the ALB SG it would be more convenient to support that change in the configuration.

Missing LogGroup creation in ecs-cluster.yaml

Hello,

is that the correct behavior that during creation of stack from ecs-cluster.yaml template, the LogGroup is not created ? In configuration, there're references to LogGroups e.g. log_group_name = ${ECSCluster}-/var/log/dmesg, but those log groups are not created.

Should they be created automatically by the instance or should I add LogGroup creation to the template ?

Regards,
Piotr

CloudWatch logs for ECS cluster hosts are not deleted when stack is deleted

The way that the CloudWatch logs (such as /var/log/dmesg) are configured for the hosts in ecs-cluster.yaml means that CF is not aware of them and therefore cannot delete them with the stack is deleted.
Unless they are created as resources in a CF template I am not sure that they can be deleted without adding a new helper script that is triggered on a delete stack.
A subsequent create using the same stack name will just re-use the existing log groups so this is not a high priority issue just a clean-up type thing (I think).

DefaultPublicRoute missing DependsOn?

Hi!

Thanks for the example, it is very helpful to me!

Should not a DependsOn on the InternetGatewayAttachment be added to the DefaultPublicRoute resource?

Indeed it looks like working better when present. in order to prevent having that type of error:

route table rtb-xxxxxxx and network gateway igw-xxxxx belong to different networks.

DefaultPublicRoute:
    Type: AWS::EC2::Route
    DependsOn: InternetGatewayAttachment
    Properties:
      RouteTableId:
        Ref: RouteTablePublic
      DestinationCidrBlock: 0.0.0.0/0
      GatewayId:
        Ref: InternetGateway

Cheers!

Repeatable

I may well be doing something wrong,
but I cannot invoke the Stack creation twice with the same name.

That is:

• I bring up the stack (Using the Launch Stack button)
• Then completely delete it.
• I can then Launch an identical stack but not if I re-use the same Stack Name.

I can't for the life of me find any dangling resources which may still be present.
This is the error message in the Cloudformation Events tab:

19:35:28 UTC-0400   ROLLBACK_IN_PROGRESS    AWS::CloudFormation::Stack  RefTest The following resource(s) failed to create: [ALB, ECS]. . Rollback requested by user.

Any Ideas?

Unused role in template

This ECSServiceAutoScalingRole does not seem to be used anywhere. Is it attached somewhere?

On the other hand for example the ECSRole is attached to the ECSInstanceProfile

Setup task role

The ECS documentation suggests best practice is to setup a task role to limit what your task can do. As far as I can see something like this should work:

    TaskRole:
        Type: AWS::IAM::Role
        Properties:
            AssumeRolePolicyDocument:
              Version: "2012-10-17"
              Statement:
                - Effect: Allow
                  Principal:
                    Service: "ecs-tasks.amazonaws.com"
                  Action:
                    - "sts:AssumeRole"
            ManagedPolicyArns:
              - "arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role"


    TaskDefinition:
        Type: AWS::ECS::TaskDefinition
        Properties:
            TaskRoleArn: !GetAtt TaskRole.Arn
            Family: product-service
            ....

Accept sub-path in listener rules

I spent way too much time debugging something that ended up being completely trivial: I had missed the fact that the example service only listen for an exact path:

    ListenerRule:
        Type: AWS::ElasticLoadBalancingV2::ListenerRule
        Properties:
            ListenerArn: !Ref Listener
            Priority: 2
            Conditions:
                - Field: path-pattern
                  Values: 
                    - !Ref Path
            Actions:
                - TargetGroupArn: !Ref TargetGroup
                  Type: forward

My services, and most I suspect unless your micro services are really tiny, handle a fair number of subpaths as well, so I needed two listeners rules:

    RootListenerRule:
        Type: AWS::ElasticLoadBalancingV2::ListenerRule
        Properties:
            ListenerArn: !Ref Listener
            Priority: 100
            Conditions:
                - Field: path-pattern
                  Values:
                    - !Ref Path
            Actions:
                - TargetGroupArn: !Ref TargetGroup
                  Type: forward

    SubListenerRule:
        Type: AWS::ElasticLoadBalancingV2::ListenerRule
        Properties:
            ListenerArn: !Ref Listener
            Priority: 101
            Conditions:
                - Field: path-pattern
                  Values:
                    - !Sub ${Path}/*
            Actions:
                - TargetGroupArn: !Ref TargetGroup
                  Type: forward

Would it make sense to do that in the reference templates as well? If so I'll happily submit a PR.