circleci-archived/enterprise-setup

Installation resources for CircleCI Enterprise

CircleCI Server Setup

This package allows you to orchestrate your CircleCI Server cluster in AWS using Terraform.

Note: This is only meant to be used for the initial setup of CircleCI Server and is not meant to be used for the ongoing maintenance of the CircleCI Server.

Note: Master is the only supported branch. All other branches of this repo should not be considered stable, and is to be used at your own risk.

Prerequisites and Requirements

Please refer to our prerequisites documentation here: https://circleci.com/docs/2.0/aws-prereq.

Installation

You can find instructions here: https://circleci.com/docs/2.0/aws/.

Variables

There are some optional variables that aren't described in the instructions. You can view their names and descriptions in variables.tf.

Var	Description	Default
services_instance_type	Instance type for the centralized services box. We recommend a m5 instance	m5.2xlarge
builder_instance_type	Instance type for the 1.0 builder machines. We recommend a r5 instance	r5.2xlarge
max_builders_count	Max number of 1.0 builders	2
nomad_client_instance_type	Instance type for the nomad clients (2.0 builders). We recommend a m5 instance	m5.2xlarge
max_clients_count	Max number of nomad clients	2
prefix	Prefix for resource names	circleci
enable_nomad	Provisions a nomad cluster for CCIE v2	1
enable_route	Enable creating a Route53 route for the Services box	0
enable_govcloud	Allows deployment into AWS GovCloud	false
route_name	Route name to configure for Services box	""
route_zone_id	Zone to configure route in	""
services_user_data_enabled	Enable/disable automated installation on Services Box	true
force_destroy_s3_bucket	Add/Remove ability to forcefully destroy S3 bucket	false
services_disable_api_termination	Protect the services instance from API termination	true

Teardown

You can find teardown instructions at https://circleci.com/docs/2.0/aws-teardown/.

Upgrade

If you want to upgrade an existing installation from a previous version, follow these upgrade instructions.

enterprise-setup's Issues

missing central-ca ami-XxxX images

variables.tf

variable "ubuntu_ami" {
  default = {
    ap-northeast-1 = "ami-2d69f14b"
    ap-northeast-2 = "ami-cd78d8a3"
    ap-southeast-1 = "ami-c38bf8bf"
    ap-southeast-2 = "ami-a437cac6"
    eu-central-1   = "ami-ff30a290"
    eu-west-1      = "ami-3cf36145"
    eu-west-2      = "ami-fd47a59a"
    sa-east-1      = "ami-24642648"
    us-east-1      = "ami-0ce3bb76"
    us-east-2      = "ami-01664c64"
    us-west-1      = "ami-98595af8"
    us-west-2      = "ami-779a2d0f"
  }
}

#89

#80

Time sync needed for services<->1.0 builder/nomad

PR submitted: #141

Request: Allow a warning status on build

Hi, my team is integrating performance metrics into the build and we would like the ability to have a job with a status of warning that turns yellow instead of just having the success/fail statuses. Is that something that can be done? I would be happy to contribute to make that happen.

Thanks!

Carrie

Device `ens3` doesn't exist and so env var PRIVATE_IP is not set.

see https://github.com/circleci/enterprise-setup/blob/master/templates/services_user_data.tpl#L20

ens3 is not always a valid device name. Without this env var set the install fails. Since this is aws specific you can use the metadata url to gain this information:

export PRIVATE_IP="$(curl http://169.254.169.254/latest/meta-data/local-ipv4)"

Feature Request: Support Multiple Environment Deployments with Terraform

I've been trying to set this up for the last few days at our company, and I've found that this repo may need a major overhaul to accomplish this. We already have a prod environment, and I went down this road because I tried to setup a staging environment for testing. I've got a working PoC internally that I can base a PR on, but essentially here is what is necessary:

1. Terraform keeps state of the current directory. As such, adding a prefix will not change anything when deploying infrastructure. In fact, it will cause some of the resources to be destroyed and recreated. We don't want this at scale if I want to have both a staging and a prod environment. Nor do I want this if I want to do something like run a CI integration that vets I can always deploy a new version to an environment by creating a test environment and then tearing it down.
2. To solve for this, all resources need to be moved from circleci.tf to modules/ folder, with each subfolder having one module per resource grouping. So, the way I have organized it in my PoC is

modules
├── aws_alb
├── aws_sqs
├── iam_instances
├── legacy-builder
├── legacy-builder-cloudinit-ubuntu-docker-v1
├── nomad
├── nomad-cloudinit-ubuntu-v1
├── s3_bucket
├── services_machine
└── vm_machine

The reason for this is so I can accomplish the next step.

3. Make variables.tf, circleci.tf, terraform.tfvars, files/, and templates/ all "templates" so to speak. Add a section to the Makefile that would allow make environment staging for example. This would then copy all of the afformentioned files to a folder structure like:

environments
├── prod
└── staging

and the sub folder structure would look like

environments/staging
├── files
├── main.tf
├── templates
├── terraform.tfstate
├── terraform.tfstate.backup
├── terraform.tfvars
└── variables.tf

And so on. This allows for me as a consumer to run terraform init/terraform plan on any environment of my choosing in the same AWS infrastructure without colliding or breaking existing infrastructure.

CircleCI 2.15

Do we need to make any changes to this terraform in order to setup CircleCI version 2.15

Request: Allow for disabling service box elastic IP by default

Currently, the enterprise setup script always generates an elastic ip for the service box. This is not something we are comfortable with since that exposes it to the entire web with quad zero security groups. It would be beneficial to add a variable to allow us to disable elastic ip assignment via the terraform script.

No enviar notificaciones a email

Po

Feature Request: Add support for disabling termination protection

Right now, things like the service box have termination enabled by default. This is great for use cases where you want to just stand up a new environment and be done with it. However, this is not great when you want to test new environment deployments. This prohibits the use of terraform destroy. It requires the user to login to AWS and manually disable those instances with Termination Protection enabled.

As per #106 , this sort of feature would go nicely with it since it would allow me to teardown and setup environments from CI for things like test cluster deployments of new infrastructure, or if I am tweaking and trying changes to a staging instance.

Add id as an output to the state file to make stopping instances easier

https://groups.google.com/forum/#!topic/terraform-tool/hEESOVOgL_Q

Not with Terraform, but it's pretty easy to stop and start an instance with the CLI tools if you have them installed:

$ aws ec2 stop-instances --region us-east-2 --instance-ids i-0123456789abcdef

$ aws ec2 start-instances --region us-east-2 --instance-ids i-0123456789abcdef

If you need to fetch the instance ID quickly, you can define a TF output and get at it that way:

$ terraform output id
i-0123456789abcdef

Or if its buried in a module somewhere you can get it from the state:

$ terraform state show module.service.aws_instance.host |awk '$1=="id"{print $3}'
i-0123456789abcdef

Combine things and do:

$ aws ec2 stop-instances --region us-east-2 --instance-ids $( terraform output id )

Enjoy.

Request: Support Setups with ALB/ELBs in front of the Service Box

Currently, due to other services not being in AWS, our team uses dnsmasq to route traffic to specific domains. dnsmasq is installed on the service box, but the service box can't be in different subnets (for obvious reasons). So, we front the service box with two ELBs. One that goes public and the other which is internal. The internal one handles a VPN connection to legacy services not in AWS. dnsmasq is then able to route traffic to these instances via the VPN. Currently, the provided script does not take this into account in anyway, and thus is a manual process. Though I have taken some cracks at it, it gets cumbersome because of things like HTTPS certs. If you set any HTTPS ports on the ELB, you need to provide a cert. Since CircleCI doesn't use certs that Amazon provides (you need to generate them then add them), this is a little more difficult to script (doable, but not impossible).

Ultimately, I'd like to see more complex use cases supported by this script if possible.

Terraform > 0.11 support

Throws warning, that will soon become errors if not addressed.

Warning: Interpolation-only expressions are deprecated

  on modules/aws_sqs/main.tf line 27, in resource "aws_iam_role_policy" "mod_role_policy":
  27:   role = "${aws_iam_role.mod_role.id}"

Terraform 0.11 and earlier required all non-constant expressions to be
provided via interpolation syntax, but this pattern is now deprecated. To
silence this warning, remove the "${ sequence from the start and the }"
sequence from the end of this expression, leaving just the inner expression.

Template interpolation syntax is still used to construct strings from
expressions when the template includes multiple interpolation sequences or a
mixture of literal strings and interpolations. This deprecation applies only
to templates that consist entirely of a single interpolation sequence.

Request: Support AWS Credentials File

Currently, the only way to run this script is with secrets. Secrets are not very useful when your company authenticates your AWS accounts via CLI and stores the AWS temp credentials locally.

Suggest changing aws_access_key and aws_secret_key to the following

variables.tf:

// variable "aws_access_key" {
//   description = "Access key used to create instances"
// }

// variable "aws_secret_key" {
//   description = "Secret key used to create instances"
// }

variable "aws_creds_file" {
  description = "AWS Credentials File"
}

variable "aws_profile" {
  description = "AWS Profile"
}

circleci.tf

provider "aws" {
  region                  = "${var.aws_region}"
  shared_credentials_file = "${var.aws_creds_file}"
  profile                 = "${var.aws_profile}"
}

terraform.tfvars

// aws_access_key = "..."
// aws_secret_key = "..."
aws_creds_file = "~/.aws/credentials"
aws_profile = "default"

As an example.

Cloudwatch Alarms and Scaling Policies exposed to end user

I began creating some Alarms, scaling policies, and what have you to ensure CircleCI workers scaled based on the threshold end users provided, but later stumbled across: https://circleci.com/docs/enterprise/cloudwatch/

It wasn't clear if CircleCI Enterprise is creating any Alarms under the hood or if it simply providing metrics in a particular namespace for monitoring health. It doesn't explicitly state that the alarms are being created, so I assume it's just the metrics (e.g ContainersAvailable).

All that said, it might be worth throwing some Alarms and scaling policies into the repo with a simple conditional for turning it on and off. Also, I would suggest exposing the threshold for said Alarms as a variable to the end user. Similarly this could be exposed for nomad cluster as well.

Something like:

resource "aws_cloudwatch_metric_alarm" "workers_out" {
  count               = "${var.enable_cw ? 1 : 0}"
  alarm_name          = "workers-scaling-out-alarm"
  comparison_operator = "LessThanThreshold"
  evaluation_periods  = "2"
  metric_name         = "ContainersAvailable"
  namespace           = "CircleCIEnterprise"
  period              = "300"
  statistic           = "Average"
  threshold           = "${var.worker_so_threshold}"

  dimensions {
    QueueName = "${aws_autoscaling_group.builder_asg.name}"
  }

  alarm_description = "This metric monitors the number containers available on the workers"
  alarm_actions     = ["${aws_autoscaling_policy.workers_out.arn}"]
}

resource "aws_autoscaling_policy" "workers_out" {
  count                  = "${var.enable_cw ? 1 : 0}"
  name                   = "workers-scaling-policy"
  scaling_adjustment     = 1
  adjustment_type        = "ChangeInCapacity"
  cooldown               = 300
  autoscaling_group_name = "${aws_autoscaling_group.builder_asg.name}"
}

Cg

Cg

Request: Remove all 0.0.0.0/0 CIDRs in favor of Security Groups/CIDR ranges

Since this is being deployed to enterprise customers, the quad zero domain is usually frowned upon as it is not very secure. Most of the traffic that needs to occur happens between the Service Box and the Builder Nodes. For this reason, some of the quad zeros can be cleaned up, and instead replaced with the subnet CIDRs or similar.

In terms of SSH capabilities, that should be a toggle honestly. The reason being not all enterprise customers will be able to utilize it depending on the security infrastructure in place. Especially at scale, it is common place to have some sort of Jump Box before hitting any infrastructure. SSH capabilities in Circle may not work because of this. Because of that, you can avoid having to handle those quad zero domains.

In addition, there should be a variable in the variables.tf file that can be set in the terraform.tfvars file for the jump box IP. This way, SSH to the service box can be limited by a CIDR range instead of being quad zero.

Pay pal

"data.template_file.output: failed to render" Error

Hi,

What I did?

I cloned the terraform from master branch and did make some changes with respect to variables according to my environment. Terraform apply failed with error "* data.template_file.output: data.template_file.output: failed to render : <template_file>:12,84-13,1: Invalid character; This character is not used within the language., and 2 other diagnostic(s)"

Troubleshooting:
Verified all files
Unable to locate data.template_file.output.

Error:

Performing plan for account aba with other options ''
export TF_DATA_DIR=.terraform.aba;cd ../enterprise-setup ; terraform plan -var-file=../Circle-Config/aba.tfvars
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

aws_security_group.circleci_builders_sg: Refreshing state... (ID: sg-0609bb05a31ccc87c)
aws_iam_role.shutdown_queue_role: Refreshing state... (ID: crcledev_shutdown_queue_role)
aws_security_group.circleci_vm_sg: Refreshing state... (ID: sg-0c099259478e7ff76)
aws_s3_bucket.circleci_bucket: Refreshing state... (ID: crcledev-bucket-829892e5)
aws_iam_role.circleci_role: Refreshing state... (ID: crcledev_role)
aws_sqs_queue.shutdown_queue: Refreshing state... (ID: https://sqs.us-east-1.amazonaws.com/306192811146/crcledev_queue)
data.aws_subnet.subnet: Refreshing state...
aws_security_group.ssh_sg: Refreshing state... (ID: sg-0e0dda648a314c4e4)
aws_iam_instance_profile.circleci_profile: Refreshing state... (ID: crcledev_profile)
data.template_file.shutdown_queue_role_policy: Refreshing state...
aws_security_group.circleci_services_sg: Refreshing state... (ID: sg-0a832ad6ca3ea1d69)
aws_iam_role_policy.shutdown_queue_role_policy: Refreshing state... (ID: crcledev_shutdown_queue_role:crcledev_shutdown_queue_role)
aws_security_group.circleci_users_sg: Refreshing state... (ID: sg-02abb562054afefa9)
aws_security_group.nomad_sg: Refreshing state... (ID: sg-07cc3df818d9d5974)
aws_security_group.circleci_builders_admin_sg: Refreshing state... (ID: sg-0f98c71983893ff58)
data.template_file.services_user_data: Refreshing state...
data.template_file.circleci_policy: Refreshing state...
aws_instance.services: Refreshing state... (ID: i-04a8dcd4081d29d18)
aws_iam_role_policy.circleci_policy: Refreshing state... (ID: crcledev_role:crcledev_policy)
data.template_file.builders_user_data: Refreshing state...
data.template_file.output: Refreshing state...
data.template_file.nomad_user_data: Refreshing state...
aws_launch_configuration.clients_lc: Refreshing state... (ID: terraform-20190507193216590600000002)
aws_launch_configuration.builder_lc: Refreshing state... (ID: terraform-20190507193216577100000001)
aws_autoscaling_group.clients_asg: Refreshing state... (ID: crcledev_nomad_clients_asg)
aws_autoscaling_group.builder_asg: Refreshing state... (ID: crcledev_builders_asg)
aws_autoscaling_lifecycle_hook.builder_shutdown_hook: Refreshing state... (ID: builder_shutdown_hook)

Error: Error refreshing state: 1 error(s) occurred:

• data.template_file.output: 1 error(s) occurred:

• data.template_file.output: data.template_file.output: failed to render : <template_file>:12,84-13,1: Invalid character; This character is not used within the language., and 2 other diagnostic(s)

It appears that terraform plan has failed

paginajuan

krintan

Cf

Ff