googlecloudplatform/solution-acceleration-toolkit

Deploy, monitor & audit on GCP simplified

Google Cloud Healthcare Data Protection Suite

Stable releases:

This repository contains a suite of tools that can be used to manage key areas of your Google Cloud organization.

• Deploy
• Monitor
• Audit

Tools

• Terraform Engine: Generate end-to-end infra-as-code for Google Cloud with security, compliance, and best practices built in.

• Policy Generator: Generate best practices policies for Forseti and other monitoring solutions, customized for your infra.

• Terraform Importer: Automatically detect and import existing resources defined by your Terraform configs.

Tutorial Video

Deploying the Data Protection Toolkit

Note that YAML-formatted configs were used at the time when the Tutorial video was made. The config format has been changed to HCL.

Releases

Please see RELEASING.md for our release strategy.

solution-acceleration-toolkit's Issues

Add integration test for policygen

Idea:

• Use Terraform Validator and fake CAI assets
• Use CFT Scorecard

Deprecate output ref

Based on #132, for dependencies it is better to be explicit on the exact path we want to depend on rather than making it confusing and trying to overdo helpers. Thus, I don't think output_ref is very valuable anymore.

#136 will help reduce duplication in output paths.

module.bigquery_destination.google_bigquery_dataset.main has change but not applied all the time

Example:

Terraform will perform the following actions:

module.bigquery_destination.google_bigquery_dataset.main will be updated in-place

~ resource "google_bigquery_dataset" "main" {
creation_time = 1586210668516
dataset_id = "1yr_org_audit_logs"
default_partition_expiration_ms = 0
default_table_expiration_ms = 31536000000
delete_contents_on_destroy = false
etag = "xsgr8/dA8yPpPQ8cHO4lZQ=="
id = "projects/<project_id>datasets/1yr_org_audit_logs"
labels = {}
last_modified_time = 1588147206214
location = "us-east1"
project = "<project_id>"
self_link = "https://www.googleapis.com/bigquery/v2/projects/<project_id>/datasets/1yr_org_audit_logs"

  - access {
      - role          = "OWNER" -> null
      - special_group = "projectOwners" -> null
    }
  + access {
      + role          = "roles/bigquery.dataOwner"
      + special_group = "projectOwners"
    }
  - access {
      - group_by_email = "[email protected]" -> null
      - role           = "READER" -> null
    }
  + access {
      + group_by_email = "[email protected]"
      + role           = "roles/bigquery.dataViewer"
    }
}

Plan: 0 to add, 1 to change, 0 to destroy.

Add hcl marshal template funcs

Support folder level audit as a component

The in the foundation recipe to choose org or folder level

Add some cron job to periodically run `terraform plan` to detect unexpected changes/upgrades.

Support plan/apply for each folder/component separately in the cloud build config.

Make --input_dir optional

We should allow users to run the importer and by default it should try to run in the cwd.

Add usage guide for policygen

Similar to https://github.com/GoogleCloudPlatform/healthcare-data-protection-suite/tree/master/cmd/tfengine#usage

Support filtering plans so the log is more readable.

Support schema for engine config and recipes

https://json-schema.org/

Add dry-run mode for importer

This mode would just print the commands it would run, instead of actually running them.

use org policy module instead of resource

Replace google_organization_policy with terraform-google-modules/terraform-google-org-policy.

Check for license headers in CI

Our CI checks should make sure all files have a license header.

Consider renaming templates to template and add label

Now that we're going to be using labels our configs currently look like this:

templates {
   ...
}

templates {
   ...
}

it is more idiomatic in Terraform to rename this to template and add a label e.g.

template "foundation" {
   ...
}

template "data_project" {
   ...
}

Look into deploying subset of the infra in CICD based on changes

Ideas:

• Use different triggers
• Use https://terragrunt.gruntwork.io/docs/reference/cli-options/#terragrunt-include-dir

Add way to reference a path from the root

Currently the dependency path is relative from the place where the config is being generated.

See if we can add a helper that can allow users to pass in paths from the root.

Remove project. and folder. prefixes from generated dirs

Better versioning in CFT Terraform container

Support project and folder level org policies.

Format and add license at the end of policygen execution.

mdcheck presubmit is now broken

Example: https://pantheon.corp.google.com/cloud-build/builds/9af7ca0a-18b8-48ff-a942-0625baf9cf00;step=0?project=hcls-data-protection-internal

Throws this error:

/usr/local/bin/markdownlint -> /usr/local/lib/node_modules/markdownlint-cli/markdownlint.js
+ [email protected]
added 36 packages from 28 contributors in 2.261s
/usr/local/lib/node_modules/markdownlint-cli/node_modules/markdownlint/lib/markdownlint.js:799
      throw error;
      ^

TypeError: content.trimEnd is not a function
    at handleRunEnd (/usr/local/lib/node_modules/markdownlint-cli/node_modules/markdownlint/lib/md037.js:38:27)
    at MD037.forEachLine (/usr/local/lib/node_modules/markdownlint-cli/node_modules/markdownlint/lib/md037.js:133:13)
    at forMetadata (/usr/local/lib/node_modules/markdownlint-cli/node_modules/markdownlint/helpers/helpers.js:256:5)
    at Array.forEach (<anonymous>)
    at forEachLine (/usr/local/lib/node_modules/markdownlint-cli/node_modules/markdownlint/helpers/helpers.js:254:16)
    at Object.MD037 [as function] (/usr/local/lib/node_modules/markdownlint-cli/node_modules/markdownlint/lib/md037.js:71:5)
    at forRule (/usr/local/lib/node_modules/markdownlint-cli/node_modules/markdownlint/lib/markdownlint.js:593:20)
    at Array.forEach (<anonymous>)
    at lintContent (/usr/local/lib/node_modules/markdownlint-cli/node_modules/markdownlint/lib/markdownlint.js:639:14)
    at lintContentWrapper (/usr/local/lib/node_modules/markdownlint-cli/node_modules/markdownlint/lib/markdownlint.js:679:12)

A new version of markdownlint-cli was released recently, 0.23.0 on May 5, 1:00am EDT (2020-05-05T04:52:03.666Z). This version is possibly broken.

Suggest to pin it to 0.22.0 for now.

Allow cicd field to be set under devops

Currently the cicd field is a sibling to devops. This requires duplicationg the devops project id and state bucket.

We should optionally allow the cicd to be set as a child of the devops field so that the common fields don't need to be duplicated.

Support child block overriding parent fields

Use case:
overriding storage bucket location on individual storage buckets or only for audit logs, etc.

Support HCL for configs and recipes

To be consistent with Terraform and Terragrunt we should also try to support hcl for all configs being written. Currently engine configs are typically written in YAML.

The primary complexity here is HCL decoder cannot decode to map[string]interface{} (hashicorp/hcl#291).

Terragrunt works around this by duplicating their structs. The input struct defines the map fields as type cty.Value and then do a manual conversion to map using helpers https://github.com/gruntwork-io/terragrunt/blob/27e0fb53c45258d183f6fbb347b718554f15dbaa/config/cty_helpers.go. Then they are set to the output struct as maps to use later on.

GKE clusters fail plan if network hasn't been deployed

GKE module also fetches the network similar to Forseti so if they are in separate deployments the plan will fail on the GKE cluster.

Add a disable for clusters so they are not initially generated.

Handle import dependencies in Importer

When importing some sets of resources, sometimes importing some resources will allow importing others. This can be handled by either:

1. Understanding the dependency graph and importing in order.
2. Retrying a full import as long as some progress is made. That is, if some resources were imported, but not all, try again, in case the successful ones now allow importing some of the previously failed ones.

Support schema in policygen

Support folder level cicd

The in the foundation recipe to choose org or folder level

Format and check hcl example/recipes

Currently only terraform .tf and terragrunt.hcl files are checked.

Support cloud dns

module "dns" {
  source  = "terraform-google-modules/cloud-dns/google"
  version = "3.0.1"

  name       = var.dns_name
  project_id = var.project_id
  type       = "public"
  domain     = var.dns_domain

  recordsets = [{
    name = "foo"
    type = "A"
    ttl  = 30
    records = [
      google_compute_global_address.foo.address,
    ]
  }]
}```

Set bool to false in TF Engine config does not actually set it in the final terraform.tfvars

Example: ENABLE_LIEN

Add support for bigquery

Set terragrunt parallelism to 1 when applying

gruntwork-io/terragrunt#1118

Synchronous deployment can be easier to debug.

Support billing account not in the same org.

Support cloud build trigger

resource "google_cloudbuild_trigger" "aaa" {
  provider = google-beta
  project  = var.project_id
  name     = "aaa"

  included_files = [
    "aaa/**"
  ]

  github {
    owner = "foo"
    name  = "bar"
    push {
      branch = "baz"
    }
  }

  filename = "aaa/cloudbuild.yaml"
}

Support notification for the post submit deployment job.

Invoke policygen functions to generate org policies in tfengine

Check/automate dependency updates

Either use Github dep bot or go get -u ./... in the presubmit

Add support for Terraform resource based Forseti policies.

Add more documentation on cicd around which folders are under trigger radar

Support recursive template definitions

This is to help reduce duplication in output paths.

e.g.

# Top level resources
- output_path: "./live"
   templates:
   ... all these templates will be under ./live ...
   - recipe_path: "../folder"
     data:
        "PROD"
   - output_path: "./folder.prod"
     templates:
      ... all these templates will be in the prod folder ...
      

Support blocking on deletion for specific resources

It would be great if we can have a whitelist or blacklist mode for resources and make the deletion check block on it.

Examples:

• in mode whitelist deletions of specific types are allowed. If a new resource wants to be deleted and allowed then first send a PR allowing that resource or resource type.
• in mode blacklist deletions of specific types are not allowed. Thus for example you can prevent any deletion of sensitive resources like GKE cluster or PHI holding resources like healthcare stores.

Add integration test for CICD components

Initial thoughts:

• Use this repo as the testing configs repo as well. Putting the configs in a testing or Terraform directory
• When cicd components change in templates, regenerate the ones in the testing config directory
• Let CICD of this repo to apply the config changes
• Deploy some dummy infra

Billing account ID and org ID should be hide in the Trigger configuration in Cloud Console and not exposed in the repo.

Add support for domain member restriction Forseti policy.

Support low risk resources such as APIs in devops project to be managed by CICD as well.

Let CICD manage APIs in devops project itself can be useful. When new resources are added in data projects, the same API needs to be enabled in the devops project in order for the Cloud Build SA to manage that resources in data projects. By supporting CICD to manage APIs in the devops project, users no longer need to deploy cicd/ directory manually just to enable APIs.

Make sure Engine pins every dependency possible.

provider, module

Add support for creating dependencies between deployments

For example the apps project needs to depend on the network project

Add folder foundation recipe

Setup CICD for toolkit

Start with just running sanity checks and go test across the repo