buildkite/on-demand

CloudFormation resources for scheduling On-Demand Buildkite Agents with AWS ECS and AWS Fargate

Important

On-Demand is no longer actively maintained or supported. We suggest exploring Agent Stack K8s as an alternative.

Buildkite On-Demand

Schedule single-shot Buildkite Agents, on-demand, on ECS.

Buildkite On-Demand is an event driven Buildkite Agent scheduler. Built on the Buildkite AWS EventBridge integration, containerised Buildkite Agents are scheduled using Amazon Elastic Container Service to run on AWS Fargate or EC2. An agent is created for each build and exits immediately on completion. There are no polling agents, you only pay for the compute platform you use.

This repository contains resources and documentation to help you configure an AWS account to schedule and run agents for your Buildkite Organization in response to builds.

Set-up Instructions

Before deploying Buildkite On-Demand to your AWS account, configure the Amazon EventBridge integration between your Buildkite and AWS accounts. See the Buildkite Documentation for how to do this.

It is best practice to run your continuous integration in a separate AWS account. Consider creating a new AWS account in your AWS Organization for the EventBridge integration.

Once you have associated your Buildkite Partner Event Source with an Amazon EventBridge bus, you are ready to deploy the on-demand stack to your AWS Account. You can deploy this stack using the AWS CloudFormation Console in a browser or the AWS CLI in your terminal.

Deploy using the AWS CloudFormation web console

The CloudFormation console will ask you for values for the following parameters:

• Stack Name: Must be unique per account per region. Use something descriptive, the default value of buildkite-on-demand is fine.
• Parameter BuildkiteAgentToken: A Buildkite Agent Registration token for this deployment to use. This will be stored in a String type AWS SSM Parameter. See the Buildkite Agent Tokens Documentation for details.
• Parameter BuildkiteQueue: The name of the Buildkite queue this stack will service. You will use this queue name in your Buildkite Pipeline Agent Query rules e.g. queue=my-queue-name.
• Parameter EventBridgeBusName: The name of an Amazon EventBridge Bus associated with a Buildkite Partner Event source NB ensure you provide the name of the EventBus name not the EventBus ARN.
• Parameter VpcSubnetIds: Optional, a comma separated list of VPC subnet IDs to schedule agents in. If left blank the default simple VPC, suitable for most use cases, will be created automatically.

When creating the stack you will need to check the option to acknowledge that the stack creates custom IAM roles.

Deploy using the AWS Serverless Application Model command line interface

The AWS SAM CLI is an extension of the AWS CLI that will auto-upload any substacks before applying your template. See the Amazon documentation for help installing the AWS SAM CLI. These instructions were written using SAM Version 0.47.0.

$ sam deploy --guided --template-file template/template.yml

This command will package and deploy the on-demand template to your AWS account, and present you with a series of prompts:

• Stack Name: The name of the CloudFormation stack to deploy. This should be unique to your account and region, something like buildkite-on-demand. You will use this stack name when updating your template in the future.
• AWS Region: The AWS region you want to deploy your on-demand agents to. Buildkite On-Demand can be deployed to multiple regions allowing you to target specific regions using Buildkite Agent Query Rules.
• Parameter EventBridgeBusName: The name of the Amazon EventBridge Bus associated with a Buildkite Partner Event source NB you provide the name of the EventBus name not the EventBus ARN.
• Parameter BuildkiteQueue: The name of the Buildkite queue this stack will service. You will use this queue name in your Buildkite Pipeline Agent Query rules e.g. queue=my-queue-name.
• Parameter BuildkiteAgentToken: A Buildkite Agent Registration token for this deployment to use. This will be stored in a String type AWS SSM Parameter. See the Buildkite Agent Tokens Documentation for details.
• Parameter VpcSubnetIds: Optional, a comma separated list of VPC subnet IDs to schedule agents in. If left blank the default simple VPC, suitable for most use cases, will be created automatically.
• Confirm changes before deploy: If set to yes, any change sets will be shown to you before execution for manual review. If set to no, the AWS SAM CLI will automatically deploy changes.
• Allow SAM CLI IAM role creation: You must answer no to this prompt to customise the capabilities.
• Capabilities [['CAPABILITY_IAM']]: Enter CAPABILITY_IAM CAPABILITY_AUTO_EXPAND to grant permission to create IAM roles and use the CloudFormation Macro to transform the agent substack.
• Save arguments to samconfig.toml: Set to yes so your choices are saved to a configuration file inside the project. In the future you can just re-run sam deploy without parameters to deploy changes.

Using ECS EC2 instead of ECS Fargate

The default on-demand template uses ECS Fargate for a simple experience, with fewer resources to operate and secure. A variant of the template that uses ECS EC2 is also available. You can launch this stack directly, though you will very likely need to fork it to suit your requirements.

You might want to use ECS EC2 for:

• control over the ECS cluster autoscaling to match your workload, scale up or down on a fixed schedule, or pay for a minimum number of instances to be present all the time
• access to larger and differentiated instances that Fargate doesn't offer
• control over Docker image caching for lower latency agent start times
• access to the host's Docker socket
• the latest EC2 features not yet available on Fargate

Modular Design

The default On-Demand template combines several components to give you a simple off the shelf experience. If you want to customise how these components are combined, you can fork the template repository.

The default template includes:

• A simple VPC with two public subnets and an Internet Gateway.
  • You can override this behaviour by providing a comma separated list of VPC Subnet IDs in the optional VpcSubnetIds parameter.
  • For more complex VPC designs or you may want to swap out the entire VPC substack with something of your own design, or !ImportValue from an existing VPC CloudFormation stack.
• A String SSM Parameter for your Buildkite Agent Registration Token.
  • CloudFormation cannot currently create SecureString parameters, if you want to store this token securely you can create it yourself and pass a different parameter path to the agent-scheduler stack.
• An agents substack and CloudFormation Macro.
  • Using a substack to define your agent task definitions and task roles ensures your infrastructure is continuously deployable. The CloudFormation Macro makes writing these task definitions easy. This component is entirely optional, you can also create your task definitions manually or using the technology stack you are most comfortable with.

Subprojects

agent-scheduler

agent-scheduler is an AWS SAM project which configures the AWS resources needed to respond to Amazon EventBridge events from Buildkite and schedule agents on ECS.

agent-composer

agent-composer is a collection of AWS CloudFormation templates to help create ECS Task Definitions that can be scheduled on-demand by agent-scheduler.

agent-composer/transform

agent-composer/transform is an AWS CloudFormation Macro that makes writing Buildkite Agent AWS::ECS::TaskDefinitions simple.

template

template is a git submodule for buildkite/on-demand-template. This is a template repository so forks are disconnected from this repository.

on-demand's Issues

Parameterise the buildkite-base and buildkite-sidecar images

These are hardcoded to keithduncan/buildkite-base and keithduncan/buildkite-sidecar repsectively.

Add support for EFS volumes in the task definitions

https://aws.amazon.com/blogs/aws/amazon-ecs-supports-efs/

Build specialised `buildkite-agent pipline upload` lambda

For first pipeline steps that clone a repository and upload a pipeline from the repo, an on-demand agent can introduce a lengthy wait. Perhaps this can be specialised by the ScheduleTask lambda to recognise simple cases and delegate them to lambda based 'specialised agents' instead.

Rework agent-scheduler VPC to support a separate VPC stack

Currently the agent-scheduler stack either creates a toy vpc, or accepts a comma separated list of vpc subnets.

Should this support taking the name of another stack + stack export name so that it can import details of the VPC instead, to prevent deleting the VPC stack while the agent-scheduler is deployed?

Use EventBridge filtering to select the interesting queue

This is done in the ScheduleTask lambda today but could be done in the EventBridge Pattern instead.

Decide whether to include the Buildkite Agent CloudFormation macro in agent-scheduler

Using the macro is very compelling, perhaps it should be deployed by default when deploying agent-scheduler to remove a step?

https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-application.html

Resources:
  AgentMacro:
    Type: AWS::Serverless::Application
    Properties:
      Location:
        ApplicationId: 'arn:aws:serverlessrepo:us-east-1:832577133680:applications/buildkite-on-demand-transform'
        SemanticVersion: '0.1.0'

Add inline IAM Policy + ManagedArn support to the Buildkite::ECS::TaskDefinition macro

If these parameters were added it would remove the main reason to create a task role outside of the Buildkite::ECS::TaskDefinition macro. The macro can still inject permission to access the given iam-ssh-agent which would remove a stumbling block when moving from the synthesized role to an explicit role.

Global resources prevent >1 deploys to the same region

agent-scheduler creates a /aws/events/Buildkite CloudWatch Log group to monitor the event bridge events being received, but this is a globally named resource and prevents deploying the scheduler to a region more than once.

The agent token is also stored globally in SSM as /buildkite/agent-token.

Parameterise the aws/ssm key id in the CloudFormation Macro

Right now the fetch + decrypt policy assumes you are using the aws/ssm key, this should be moved to the Globals section of the transform as a default value and individual Secret entries should allow override.

Access Denied on S3 template URL

We are trying to deploy this stack and are currently seeing an Access Denied message for the following S3 template URL's.

• https://s3.us-east-1.amazonaws.com/buildkite-serverless-apps-us-east-1/on-demand/template/695ea39a20031481dad67f7caaa1bfaa.template

• https://s3.us-east-1.amazonaws.com/buildkite-serverless-apps-us-east-1/on-demand/template/680ce2f16a1e7f87905f2452398904ea.template

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>P24WHP9ME7CYMW2B</RequestId>
<HostId>
yt7S1rUlHWRn8o5jyWOUXMr3exTyq6lUqu+Ed9naFflvW4YzGSl6DjO6rsc6THTxQRejogFKcrg=
</HostId>
</Error>

Try using SSM parameter for IAM SSH Agent backend ARN

Allow the stack to passively discover an IAM SSH Agent backend configured using SSM.

The IAM SSH Agent ARN is needed in several places, CloudFormation parameters, dynamic task definition generation. It might make sense to make this an SSM parameter that is set once when deploying IAM SSH Agent for your infrastructure, and all the components auto discover it.

Putting the stack name in the parameter would ensure multiple deployments in the same region can use or not use iam ssh agent.

Make the IAM TaskRole and ExecutionRole prefix stack-name specific

To facilitate separate queues having access to different IAM roles in an account, the iam:PassRole grant can be scoped to a path scoped to each agent-scheduler that is deployed. This is needed because IAM is a global resource but agent-scheduler can be deployed more than once.

These paths need to be generated by the agent-scheduler and passed to the existing resources. They will also need to be consumed by agent-composer stacks to place IAM roles in an appropriate spot.

This could be done with an agent-scheduler stack export and by passing the agent-scheduler stack name in to the agent-composer sub stack.

Add support for Fargate Spot

Depends on capacity provider support in CloudFormation aws/containers-roadmap#631