ovotech/cloud-key-rotator

A Golang program to rotate AWS & GCP account keys

Cloud-Key-Rotator

⚠️ this is no longer supported. Security fixes will continue to be applied, but if you need any other changes, please fork.

This is a Golang program to assist with the reporting of Service Account key ages, and rotating said keys once they pass a specific age threshold.

The tool can update keys held in the following locations:

• Atlas (mongoDB)
• CircleCI env vars
• CircleCI contexts
• Datadog (GCP Integration)
• GCS
• Git
• GitHub Secrets
• GoCd
• K8S (GKE only)
• SSM (AWS Parameter Store)
• AWS SecretsManager

The tool is packaged as an executable file for native invocation, and as a zip file for deployment as an AWS Lambda.

ℹ️ where possible OpenID Connect (OIDC) should be used instead of furnishing/storing long-lived credentials. Using OIDC will remove the need for running cloud-key-rotator.

Install

From Binary Releases

Darwin, Linux and Windows Binaries can be downloaded from the Releases page.

Try it out:

$ cloud-key-rotator -h

Docker Image

An Alpine-based Docker image is available here.

Getting Started

Config

cloud-key-rotator picks up details about which key(s) to rotate, and locations to update with new keys, from config.

Check out examples for example config files. Viper is used as the config framework, so config can be stored as JSON, TOML, YAML or HCL.

For native invocation, the file needs to be called "config" (before whatever extension you're using), and be present either in /etc/cloud-key-rotator/ or in the same directory the binary runs in. For AWS Lambda invocation, the config needs to be set as a plaintext secret in the AWS Secrets Manager, using a default key name of "ckr-config".

Authentication/Authorisation

You'll need to provide cloud-key-rotator with the means of authenticating into any key provider that it'll be updating.

Authorisation is handled by the Default Credential Provider Chains for both GCP and AWS.

Mode Of Operation

cloud-key-rotator can operate in two different modes:

1. Rotation mode - in which keys are rotated; and
2. Non-rotation mode - which only posts the ages of keys to the Datadog Metric API.

The boolean field RotationMode config controls the mode of operation.

Age Thresholds

You can set the age threshold to whatever you want in the config, using the DefaultRotationAgeThresholdMins field in config, or you can override on a per-service-account-basis with the RotationAgeThresholdMins field. Key ages are always measured in minutes.

cloud-key-rotator will not attempt to rotate a key until it's passed the age threshold you've set (either default or the key-specific). This allows you to run the tool as frequently as you want without worrying about keys being rotated excessively.

Key Locations

"Key locations" is the term used for the places where keys are stored, which will ultimately be updated with the new keys that are generated.

Currently, the following locations are supported:

• Atlas (mongoDB)
• CircleCI env vars
• CircleCI contexts
• Datadog (GCP Integration)
• GCS
• Git (files encrypted with mantle which integrates with KMS))
• GitHub Secrets
• GoCd
• K8S (GKE only)
• SSM (AWS Parameter Store)
• AWS SecretsManager

Rotation Process

The tool attempts to verify its actions as much as possible and aborts immediately if it encounters an error. By design, the tool does not attempt to handle errors gracefully and continue, since this can lead to a "split-brain effect", with keys out-of-sync in various locations.

It should be quick to re-run the tool (with new keys being created) once issues have been resolved. Note that cloud providers usually limit the number of keys you can have attached to a Service Account at any one time, so it is worth bearing this in mind when re-running manually after seeing errors.

Only the first key of a Service Account is handled by cloud-key-rotator. If it handled more than one key, it could lead to complications when updating single sources multiple times.

Key Sources

The AccountKeyLocations section of config holds details of the places where the keys are stored, e.g.:

"AccountKeyLocations": [{
  "ServiceAccountName": "cloud-key-client-test",
  "RotationAgeThresholdMins": 60,
  "Git": {
    "FilePath": "service-account.txt",
    "OrgRepo": "ovotech/cloud-key-rotator",
    "VerifyCircleCISuccess": true,
    "CircleCIDeployJobName": "dummy_deploy_with_wait"
  },
  "CircleCI": [{
    "UsernameProject": "ovotech/cloud-key-rotator",
    "KeyEnvVar": "ENV_VAR_NAME"
  }],
  "K8s": [{
    "Project": "my_project",
    "Location": "europe-west2-b",
    "ClusterName": "cluster_name",
    "Namespace": "uat",
    "SecretName": "key-rotate-test-secret",
    "DataName": "my-key.json"
  }]
}]

cloud-key-rotator has integrations into GitHub and CircleCI, which allows it not only to update those sources with the new key, but also to verify that a deployment has been successful after committing to a GitHub repository. If that verification isn't required, you can disable it using the VerifyCircleCISuccess boolean.

For any Git key location, the whole process will be aborted if there is no KmsKey value set. Unencrypted keys should never be committed to a Git repository.

GPG Commit Signing

Commits to Git repositories are required to be GPG signed. In order to achieve this, you need to provide 4 things:

• Username of the Git user commits will be made on behalf of, set in config
• Email address of Git user, set in config
• ArmouredKeyRing, aka GPG private key, stored in /etc/cloud-key-rotator/akr.asc
• Passphrase to the ArmouredKeyRing

e.g. along with the akr.asc file, you should set the following:

"AkrPass": "change_me",
"GitName": "git-name",
"GitEmail": "[email protected]",

Filtering Service Accounts

You may want to only include or exclude specific Service Accounts. This is possible using AccountFilter.

E.g.:

"AccountFilter": {
  "Mode": "include",
  "Accounts": [{
    "Provider": {
      "Name": "gcp",
      "Project": "my-project"
    },
    "ProviderAccounts": [
      "cloud-key-client-test"
    ]
  }]
}

Mode field is either include or exclude. If you omit the AccountFilter, the rotation process will fail.

Notice how the name of the Service Account is used. In the case of GCP, this is everything preceding the @[project].iam.gserviceaccount.com string in the Service Account's email address.

Rotation Flow

1. Reduce keys to those of service accounts deemed to be valid (e.g. strip out user accounts if in rotation-mode)
2. Filter keys to those deemed to be eligible (e.g. according to filtering rules configured by the user)
3. For each eligible key:

• Create new key
• Update key locations
• Verify update has worked (where possible)
• Delete old key

Troubleshooting

GCP

• I get an error when using my key: Response: {"error":"invalid_grant", "error_description":"Invalid JWT Signature."}

The key you're trying to use isn't valid. It could be because a rotation has happened since the process, e.g. CI/CD job, started using the key (hence the key has been deleted in GCP). If you run the cloud-key-rotator very frequently it increases your chance of seeing this.

Try and schedule rotations for times that are unlikely to conflict with CI/CD jobs.

• When trying to create a GCP AppEngine App (a pre-requisite of being able to create CloudScheduler jobs) I get an error: Error waiting for App Engine app to create: Error code 13, message: AppEngine service account cannot be generated for e~<project_name>

This happens when the AppEngine default service account has been previously deleted from your project. If this happeed recently (possibly last 30d) there's a gcloud cmd available to try. If that's not possible, GCP support should be able to restore the service account.

• CloudScheduler fails to invoke the cloud-key-rotator CloudFunction, showing a PERMISSION DENIED error in logs

For the CloudScheduler to have permission to invoke CloudFunctions, the Cloud Scheduler service account must be given the Cloud Scheduler Service Agent role in your project IAMs.

The Cloud Scheduler service account will have an id of format:

service-<project_number>@gcp-sa-cloudscheduler.iam.gserviceaccount.com

CircleCI

• I get an error in cloud-key-rotator logs: 404: Project not found: APIError null

First thing to check is that there's no typo in the UsernameProject value that you've set in config.

The user/owner (preferably a bot user) of the CircleCI API key that you pass to cloud-key-rotator must have write access to the GitHub repo that the CircleCI jobs run from.

Contributions

Contributions are more than welcome from both internal (to ovotech) and external contributors.

If you have write access to this repo, create a branch and PR, otherwise fork and PR. Forked branches will be pushed to this repo by a reviewer so PR checks can run.

cloud-key-rotator's Issues

Add ability to override age threshold per SA

This would allow users to scale down their key lifetimes on a per SA/key basis, which may be more feasible for them than doing the whole lot in one go.

Invalid config doesn't error in GCP CloudFunction

We just get the log output "Filtered down to 0 keys based on current app config" when the config is invalid.

I would have expected viper to return an error when unmarshalling

https://github.com/ovotech/cloud-key-rotator/blob/d777057d6583a843b7d3be6cfdc9e190e37a665d/pkg/config/config.go#L127:L149

//GetConfigFromGCS grabs the cloud-key-rotator's config from GCS
func GetConfigFromGCS(bucketName, objectName, configType string) (c Config, err error) {
	ctx := context.Background()
	var client *storage.Client
	if client, err = storage.NewClient(ctx); err != nil {
		return
	}
	bkt := client.Bucket(bucketName)
	obj := bkt.Object(objectName)
	var rc *storage.Reader
	if rc, err = obj.NewReader(ctx); err != nil {
		return
	}
	defer rc.Close()
	var data []byte
	if data, err = ioutil.ReadAll(rc); err != nil {
		return
	}
	viper.SetConfigType(configType)
	viper.ReadConfig(bytes.NewReader(data))
	err = viper.Unmarshal(&c)
	return
}

Test improvement proposal

• mock call to ovotech/cloud-key-client

• local key writer

• Unit tests public Rotate() func

• Add e2e tests for all key writers with real resources

Output key ages to logs

...not just as a metric to Datadog.

Add Google KMS support

Add e2e test

Add GOCD location type

there are some GOCD clients written in Go already, e.g.: https://github.com/beamly/go-gocd

Mock out locations and create tests

There's currently no testing of locations, it'd be great if we could mock them out (since they all involve calls to external APIs) and get some proper testing on them.

Allow pause time after key sources have been updated

....in case users want to bail out of the key deletion.

Add GCS location

Add config file functionality

It'd be nice to be able to pull config from a file, rather than env vars

Add 'rotate all' setting

Currently, if a user wants to rotate ALL their SA keys, they won't be able to without specifying each service account separately, as for safety, cloud-key-rotator doesn't rotate anything when neither include or exclude SAs are set.

Users should be able to rotate ALL their keys if they want to.

Add json output

Add PR check for whether `go mod tidy` produces a diff

I think go build will need to be checked in the same way

The goreleaser job in circleci is quite flakey, I'm sure there's various reasons, including go versions being used by people locally that doesn't match that of the circleci jobs.

If after running go mod tidy or go build there are changes to go.mod or go.sum, the check should fail (as eventually go releaser would end up failing when a new tag is pushed and the release circleci job runs.

Add key 'dater' functionality to CLI

...it would allow users to see the ages of their keys from command line. 🎉

Add Lambda program model

https://docs.aws.amazon.com/lambda/latest/dg/go-programming-model-handler-types.html

should be able to package it up and release as a .zip as well, so users can deploy as Lambdas easily.

TF Module: Recreate lambda when binary version is changed

Still need to make sure that terraform knows the binary has changed on update: https://www.terraform.io/docs/providers/aws/r/lambda_function.html#source_code_hash perhaps?

Can we find a way to set the latest binary version name as a default in a variable?

Add publish lambda.zip to s3 on new release

Currently we're not publishing new lambda.zip artifacts to s3. This needs to be done so users of the terraform module can specify new versions.

Handle AWS vs. GCP keys more gracefully

Differences in logic in handling AWS vs. GCP keys exists in multiple places, and not very gracefully, e.g. deciding whether a key needs to be base64 decoded or not.

It'd be great if this could be specified on a provider level.

Add 'ignore SA' functionality

Could accept a list via config/env var?

Add Kubernetes source

Some keys may be used in Kubernetes, so that'll need adding as a 'source' so cloud-key-rotator can update them with new keys

Add alternative path for armoured keyring

When users are running the tool locally, they probably won't want to have to store the armoured keyring in /etc/cloud-key-rotator

They should either be able to set an alternative path, or in pwd

Account field is blank in logs

e.g.:

{
  "level": "info",
  "ts": 1559750287.9522605,
  "caller": "rotate/rotatekeys.go:273",
  "msg": "Key locations updated",
  "keyProvider": "gcp",
  "account": "",
  "keyID": "**changed**",
  "keyLocationUpdates": [
    {
      "LocationType": "GitHub",
      "LocationURI": "**changed**",
      "LocationIDs": [
        "**changed**"
      ]
    }
  ]
}

Default env var names for AWS keys

AWS key locations that are env vars will almost always have these env var names by default:

      "KeyIDEnvVar": "AWS_ACCESS_KEY_ID",
      "KeyEnvVar": "AWS_SECRET_ACCESS_KEY"

we could use those names as default, too, so users won't have to explicitly set them in config in most cases.

Switch on validation of encrypted ciphertexts

...right now this would only apply to keys stored in GitHub, as that's the only time encryption is needed.

ovotech/mantle has the ability to validate ciphertexts for us, so we'd just need to switch it on here:

cloud-key-rotator/pkg/crypt/crypt.go

Line 30 in b1aae8e

const disableValidation = true

I can't remember the original reason why this was disabled, so thorough testing required.

Add key rotation functionality

This will need:

• gcloud integration to create new key + delete old one
• aes-256-gcm-kms reencrypt capability
• git integration (to update the source file)

Use login-profile existence on AWS users to determine whether user is human

in rotatekeys.go, we currently use regex to determine whether a user is human or not:

//validAwsKey returns a bool that reflects whether the provided keys.Key is
// valid, based on aws-specific rules
func validAwsKey(key keys.Key, config config.Config) (valid bool) {
	if config.IncludeAwsUserKeys {
		valid = true
	} else {
		match, _ := regexp.MatchString("[a-zA-Z]\\.[a-zA-Z]", key.Name)
		valid = !match
	}
	return
}

we should instead determine this from the existence of a login-profile for the user. Potentially this

Add support for rotating GCP in Lambda

Users who want to rotate their GCP service account keys while running CKR in a Lambda will need to have the GCP key provisioned somehow.

Should be possible to do this by downloading a key.json from AWS Secrets Manager (there's already a func that determines if running in Lambda)

Enable GCP APIs via terraform

To use the GCP terraform module, the following APIs need to be enabled:

• cloudbuild.googleapis.com
• cloudscheduler.googleapis.com
• cloudfunctions.googleapis.com
• appengine.googleapis.com

it'd be great to have these enabled via the terraform module itself.

Allow manual invocation of only one SA key rotation

Add generic 'validateKeys' func

Any specifics on performing specific validation depending on the provider can then be done within the validateKeys() func, rather than have this specific branching within the filterKeys() func

Add self-awareness

The cloud-key-rotator should be able to rotate the key that it's using to perform the rotation process. Ideally this would be the last of all the rotations handled in a process, otherwise the application would have to pick up the key change on the fly

Add key rotation threshold in skip message

Add key rotation threshold to output of:

Skipping SA: xxxx, key: xxxxxxxxxxx as it's only x minutes old.

Add AWS parameter store location type

Add atlas/mongodb integration

..so that AWS keys can be rotated + updated in atlas/mongodb

Improve examples

The existing examples can do with a revamp. I think it'd be nice to have a set of minimal examples in a README.md

Add config validate & generate cmds

...could be useful on command-line to quickly check new config is valid.

Add AWS creds file composition

'Service Account' credentials for AWS are returned in the form of Access key ID and Secret access key.

Users may want to update locations with these creds in the form of a credentials file, as opposed to using env vars.

Only auth against required providers

E.g., if specifying ./cloud-key-rotator rotate -a my-gcp-sa, currently cloud-key-rotator will initially try obtain key details for all providers enabled in config, whereas it only needs to do this for GCP (in this example).

As a result of this ^^, if I haven't set the AWS auth token, I get the error:

{"level":"error",
"ts":1554199286.953519,
"caller":"cmd/rotate.go:113",
"msg":"ExpiredToken: The security token included in the request is expired\n\tstatus code: 403, request id: 47916c90-552e-11e9-b21c-458735d46cc9","stacktrace":"github.com/ovotech/cloud-key-rotator/cmd.glob..func1\n\t/Users/chris.every/go/src/github.com/ovotech/cloud-key-rotator/cmd/rotate.go:113\ngithub.com/spf13/cobra.(*Command).execute\n\t/Users/chris.every/go/src/github.com/spf13/cobra/command.go:766\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/Users/chris.every/go/src/github.com/spf13/cobra/command.go:852\ngithub.com/spf13/cobra.(*Command).Execute\n\t/Users/chris.every/go/src/github.com/spf13/cobra/command.go:800\ngithub.com/ovotech/cloud-key-rotator/cmd.Execute\n\t/Users/chris.every/go/src/github.com/ovotech/cloud-key-rotator/cmd/root.go:20\nmain.main\n\t/Users/chris.every/go/src/github.com/ovotech/cloud-key-rotator/main.go:8\nruntime.main\n\t/usr/local/Cellar/go/1.11/libexec/src/runtime/proc.go:201"}

Trim whitespace from user/service-account names

This has burnt me a few times when adding new AWS users to config. From copy/pasting from the AWS UI there's a trailing space depending where you copy from

Add minimum set of priviliges for GCP

for key dating, the minimum set of perms is:

    iam.serviceAccountKeys.list
    iam.serviceAccounts.list

it'd be good to get this documented (and the same for full rotation mode).

Support JSON in GCS output

Prevent GCP key rotation from starting if service account already has 10 keys

...10 keys is the limit enforced by GCP

Mask chars of key Ids

It's probably best we don't log key Ids out, but it'd still be nice to be able to identify them. Perhaps just mask all but the last 4 chars?

Pipeline doesn't upload releases to S3

The CircleCI pipeline doesn't upload releases to the public S3 bucket. Currently I can only see one release in the bucket which I assume was uploaded manually.

This causes the Terraform module to fail when applying any plan that specifies any ckr_version other than 0.27.18

https://github.com/ovotech/cloud-key-rotator/blob/master/tf_module/ckr/main.tf#L107

Generecise GitHub integration

cloud-key-rotator should be agnostic to the Git provider. E.g., it should be possible to use GitHub OR GitLab.

Add version cmd

It'd also be great to have the current version logged at the start of a rotation process.

Add static code analysis

• go_build
• go_cyclo
• go_fmt
• go_lint
• go_vet

Add unit tests

Add prometheus endpoint for key ages