osinfra-io/terraform-datadog-google-integration

Terraform example module for Datadog Google Cloud Platform integration.

Datadog - Google Cloud Platform Integration Terraform Module

GitHub Actions:

Infracost:

💵 Monthly estimates based on Infracost baseline costs.

Repository Description

Terraform example module for Datadog Google Cloud Platform integration.

Note

We do not recommend consuming this module like you might a public module. It is a baseline, something you can fork, potentially maintain, and modify to fit your organization's needs. Using public modules vs. writing your own has various drivers and trade-offs that your organization should evaluate.

🔩 Usage

Tip

You can check the fixtures directory for example configurations. These fixtures set up the system for testing by providing all the necessary initial code, thus creating good examples on which to base your configurations.

Required APIs (managed with the terraform-google-project child module):

• bigquerydatatransfer.googleapis.com (If enable_cloud_cost_management is true)
• bigquery.googleapis.com (If enable_cloud_cost_management is true)
• cloudasset.googleapis.com
• cloudbilling.googleapis.com
• cloudresourcemanager.googleapis.com
• compute.googleapis.com
• iam.googleapis.com
• monitoring.googleapis.com

Development

Our focus is on the core fundamental practice of platform engineering, Infrastructure as Code.

Open Source Infrastructure (as Code) is a development model for infrastructure that focuses on open collaboration and applying relative lessons learned from software development practices that organizations can use internally at scale. - Open Source Infrastructure (as Code)

To avoid slowing down stream-aligned teams, we want to open up the possibility for contributions. The Open Source Infrastructure (as Code) model allows team members external to the platform team to contribute with only a slight increase in cognitive load. This section is for developers who want to contribute to this repository, describing the tools used, the skills, and the knowledge required, along with Terraform documentation.

See the documentation for setting up a local development environment here.

🛠️ Tools

• infracost
• pre-commit
• pre-commit-terraform
• terraform-docs

📋 Skills and Knowledge

Links to documentation and other resources required to develop and iterate in this repository successfully.

• gcp-cloud-cost-management
• gcp-integration

🔍 Tests

You'll need to be a member of the platform-contributors Google Group to run the tests. This group manages access to the resource hierarchy's Testing/Sandbox folder. You can request access to this group by opening an issue here.

cd fixtures/default

terraform init

terraform test -var="api_key=$DATADOG_API_KEY" -var="app_key=$DATADOG_APP_KEY"

📓 Terraform Documentation

• global

terraform-datadog-google-integration's Issues

Support for exclusion filters

Log entries that match any of the exclusion filters will not be exported.

Tasks

Beta Give feedback

1. Can add custom exclusions

• https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/logging_project_sink#nested_exclusions
• https://cloud.google.com/logging/docs/view/building-queries

Resource:
cloudresourcemanager.googleapis.com
monitoring.googleapis.com

Events:

• google.monitoring.v3.MetricService.ListTimeSeries
• google.monitoring.v3.MetricService.ListMetricDescriptors

IDs:

• datadog@*.iam.gserviceaccount.com

Deprecation of Kitchen-Terraform

The official announcement from Aaron Lane:

With the announcement of a native test framework with Terraform 1.6 on 2023-10-04, we are excited/saddened/relieved to announce the deprecation of Kitchen-Terraform 🎉 The project will enter a maintenance-only period for about 1 year, after which time it will be archived.

Tasks

Beta Give feedback

1. Tests re-written using native test framework

Don't run Kitchen Test workflow if GitHub user is osinfra-sa

Dependabot manages dependencies for bundler and GitHub Actions. Dependabot opens the PR; however, the osinfra-sa users merges, triggering a workflow we do not want to run.

Tasks

Beta Give feedback

1. Dependabot triggered Kitchen Test workflows do not run

Example PR:

• https://github.com/osinfra-io/terraform-google-kubernetes-engine/pull/12/files

if: GitHub.actor != 'dependabot[bot]' 

Enable Infracost template and sync usage file with pre-commit

We learned we could simplify developers' management of usage files.

Tasks

Beta Give feedback

1. Add Infracost template
2. Pre-commit should sync usage file

• https://www.infracost.io/docs/features/config_file

BigQuery data sets should specify a default customer-managed encryption key

By default, BigQuery uses envelope encryption with Google-managed cryptographic keys to encrypt the data at rest. The data is encrypted using data encryption keys, and the data encryption keys themselves are further encrypted using key encryption keys. This is seamless and does not require any additional input from the user. For greater control, customer-managed encryption keys (CMEKs) can be used as an encryption key management solution for BigQuery datasets.

Enable Google Cloud Armor metrics, events, and logs in Datadog

Enable this integration along with the Google Cloud Security Command Center Integration to visualize DDoS threats to your Google Cloud environment in Datadog. With this integration, Datadog collects important security events from your Google Cloud network security configurations and metrics from Google Cloud Armor.

Force cost-center label

Something like: https://github.com/osinfra-io/terraform-google-project/blob/main/global/main.tf#L228

BigQuery tables should be encrypted with customer-managed encryption keys

By default, BigQuery encrypts data at rest by employing Envelope Encryption using Google managed cryptographic keys. The data is encrypted using the data encryption keys and data encryption keys themselves are further encrypted using key encryption keys. This is done automatically and does not require any additional input from the user. However, if you want to have greater control, customer-managed encryption keys (CMEK) can be used as an encryption key management solution for BigQuery datasets. If CMEK is used, the CMEK is used to encrypt the data encryption keys, instead of using google-managed encryption keys.

Support for enabling Cloud Cost Management

Grant access to schedule exporting cost data from BigQuery to Cloud Storage, so that you can perform analysis on cloud costs and usage data across Datadog. This setup process (documentation) requires that the Google Cloud Integration be enabled on a project.

Tasks

Beta Give feedback

1. Add BigQuery dataset for cloud cost management #13
   +0
   
2. Add bucket and IAM roles for cloud cost management #14
   +0
   

Replace `bigquery.admin` with custom role

    "roles/bigquery.admin", # TODO: Replace with custom role

terraform-datadog-google-integration/global/main.tf

Line 97 in c9b330f

# https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/google_project_iam#google_project_iam_member

Initial development

Empty tuple for `google_bigquery_dataset.billing_export`

│ Error: Invalid index
│ 
│   on .terraform/modules/datadog/global/main.tf line 40, in resource "google_bigquery_dataset_iam_member" "billing_export":
│   40:   dataset_id = google_bigquery_dataset.billing_export[0].dataset_id
│     ├────────────────
│     │ google_bigquery_dataset.billing_export is empty tuple
│ 
│ The given key does not identify an element in this collection value: the
│ collection has no elements.