Creates opinionated BigQuery datasets and tables
Home Page: https://registry.terraform.io/modules/terraform-google-modules/bigquery/google
License: Apache License 2.0
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
![release-please[bot] avatar](https://avatars.githubusercontent.com/in/40688?v=4 "release-please[bot]")
![renovate[bot] avatar](https://avatars.githubusercontent.com/in/2740?v=4 "renovate[bot]")
Items found while testing PR #92.
I have tested from master, with the same symptoms and failures. I can roll in changes for item 1 to my PR if that's appropriate, but have not solved item 2. Also not sure / wondering if CI run failures on the same PR are same / similar to item 2.
kpeder@kpeder-develop:~/projects/terraform-google-bigquery$ git diff
diff --git a/test/setup/main.tf b/test/setup/main.tf
index eab6c3c..4bcd739 100644
--- a/test/setup/main.tf
+++ b/test/setup/main.tf
@@ -16,7 +16,7 @@
module "project" {
source = "terraform-google-modules/project-factory/google"
- version = "~> 3.0"
+ version = "~> 9.2.0"
name = "ci-bigquery"
random_project_id = "true"
diff --git a/test/setup/versions.tf b/test/setup/versions.tf
index efbd8ea..a3ffc24 100644
--- a/test/setup/versions.tf
+++ b/test/setup/versions.tf
@@ -19,9 +19,9 @@ terraform {
}
provider "google" {
- version = "~> 2.13.0"
+ version = "~> 3.41.0"
}
provider "google-beta" {
- version = "~> 2.13.0"
+ version = "~> 3.41.0"
}
I am pasting the test run here in the hope that it can be resolved by a smart person who knows more about kitchen and ruby than I.
kpeder@kpeder-develop:~/projects/terraform-google-bigquery$ make docker_test_integration
docker run --rm -it \
-e SERVICE_ACCOUNT_JSON \
-v /home/kpeder/projects/terraform-google-bigquery:/workspace \
gcr.io/cloud-foundation-cicd/cft/developer-tools:0 \
/usr/local/bin/test_integration.sh
Activated service account credentials for: [[email protected]]
Automatically setting inputs from outputs of test/setup
Activated service account credentials for: [[email protected]]
-----> Starting Test Kitchen (v2.7.2)
-----> Creating <full-local>...
$$$$$$ Verifying the Terraform client version is in the supported interval of >= 0.11.4, < 0.14.0...
$$$$$$ Reading the Terraform client version...
Terraform v0.13.5
+ provider registry.terraform.io/hashicorp/external v2.0.0
+ provider registry.terraform.io/hashicorp/google v3.47.0
+ provider registry.terraform.io/hashicorp/null v3.0.0
+ provider registry.terraform.io/hashicorp/random v3.0.0
$$$$$$ Finished reading the Terraform client version.
$$$$$$ Finished verifying the Terraform client version.
$$$$$$ Initializing the Terraform working directory...
Upgrading modules...
- example in ../../../examples/multiple_tables
- example.add_udfs in ../../../modules/udf
Downloading github.com/terraform-google-modules/terraform-google-gcloud for example.add_udfs.bq_check_protocol...
- example.add_udfs.bq_check_protocol in .terraform/modules/example.add_udfs.bq_check_protocol
Downloading github.com/terraform-google-modules/terraform-google-gcloud for example.add_udfs.bq_csv_to_struct...
- example.add_udfs.bq_csv_to_struct in .terraform/modules/example.add_udfs.bq_csv_to_struct
Downloading github.com/terraform-google-modules/terraform-google-gcloud for example.add_udfs.bq_find_in_set...
- example.add_udfs.bq_find_in_set in .terraform/modules/example.add_udfs.bq_find_in_set
Downloading github.com/terraform-google-modules/terraform-google-gcloud for example.add_udfs.bq_parse_url...
- example.add_udfs.bq_parse_url in .terraform/modules/example.add_udfs.bq_parse_url
- example.bigquery in ../../..
Initializing the backend...
Initializing provider plugins...
- Finding latest version of hashicorp/external...
- Finding hashicorp/google versions matching "~> 3.0"...
- Finding latest version of hashicorp/null...
- Finding latest version of hashicorp/random...
- Installing hashicorp/google v3.47.0...
- Installed hashicorp/google v3.47.0 (signed by HashiCorp)
- Installing hashicorp/null v3.0.0...
- Installed hashicorp/null v3.0.0 (signed by HashiCorp)
- Installing hashicorp/random v3.0.0...
- Installed hashicorp/random v3.0.0 (signed by HashiCorp)
- Installing hashicorp/external v2.0.0...
- Installed hashicorp/external v2.0.0 (signed by HashiCorp)
The following providers do not have any version constraints in configuration,
so the latest version was installed.
To prevent automatic upgrades to new major versions that may contain breaking
changes, we recommend adding version constraints in a required_providers block
in your configuration, with the constraint strings suggested below.
* hashicorp/external: version = "~> 2.0.0"
* hashicorp/null: version = "~> 3.0.0"
* hashicorp/random: version = "~> 3.0.0"
Terraform has been successfully initialized!
$$$$$$ Finished initializing the Terraform working directory.
$$$$$$ Creating the kitchen-terraform-full-local Terraform workspace...
Created and switched to workspace "kitchen-terraform-full-local"!
You're now on a new, empty workspace. Workspaces isolate their state,
so if you run "terraform plan" Terraform will not see any existing state
for this configuration.
$$$$$$ Finished creating the kitchen-terraform-full-local Terraform workspace.
Finished creating <full-local> (0m24.41s).
-----> Test Kitchen is finished. (0m27.69s)
-----> Starting Test Kitchen (v2.7.2)
-----> Converging <full-local>...
$$$$$$ Verifying the Terraform client version is in the supported interval of >= 0.11.4, < 0.14.0...
$$$$$$ Reading the Terraform client version...
Terraform v0.13.5
+ provider registry.terraform.io/hashicorp/external v2.0.0
+ provider registry.terraform.io/hashicorp/google v3.47.0
+ provider registry.terraform.io/hashicorp/null v3.0.0
+ provider registry.terraform.io/hashicorp/random v3.0.0
$$$$$$ Finished reading the Terraform client version.
$$$$$$ Finished verifying the Terraform client version.
$$$$$$ Selecting the kitchen-terraform-full-local Terraform workspace...
$$$$$$ Finished selecting the kitchen-terraform-full-local Terraform workspace.
$$$$$$ Downloading the modules needed for the Terraform configuration...
- example in ../../../examples/multiple_tables
- example.add_udfs in ../../../modules/udf
Downloading github.com/terraform-google-modules/terraform-google-gcloud for example.add_udfs.bq_check_protocol...
- example.add_udfs.bq_check_protocol in .terraform/modules/example.add_udfs.bq_check_protocol
Downloading github.com/terraform-google-modules/terraform-google-gcloud for example.add_udfs.bq_csv_to_struct...
- example.add_udfs.bq_csv_to_struct in .terraform/modules/example.add_udfs.bq_csv_to_struct
Downloading github.com/terraform-google-modules/terraform-google-gcloud for example.add_udfs.bq_find_in_set...
- example.add_udfs.bq_find_in_set in .terraform/modules/example.add_udfs.bq_find_in_set
Downloading github.com/terraform-google-modules/terraform-google-gcloud for example.add_udfs.bq_parse_url...
- example.add_udfs.bq_parse_url in .terraform/modules/example.add_udfs.bq_parse_url
- example.bigquery in ../../..
$$$$$$ Finished downloading the modules needed for the Terraform configuration.
$$$$$$ Validating the Terraform configuration files...
Success! The configuration is valid.
$$$$$$ Finished validating the Terraform configuration files.
$$$$$$ Building the infrastructure based on the Terraform configuration...
module.example.module.bigquery.google_bigquery_dataset.main: Creating...
module.example.module.bigquery.google_bigquery_dataset.main: Creation complete after 1s [id=projects/ci-bigquery-76f4/datasets/foo]
module.example.module.bigquery.google_bigquery_table.main["bar"]: Creating...
module.example.module.bigquery.google_bigquery_table.main["foo"]: Creating...
module.example.module.bigquery.google_bigquery_table.main["foo"]: Creation complete after 1s [id=projects/ci-bigquery-76f4/datasets/foo/tables/foo]
module.example.module.bigquery.google_bigquery_table.main["bar"]: Creation complete after 1s [id=projects/ci-bigquery-76f4/datasets/foo/tables/bar]
Apply complete! Resources: 3 added, 0 changed, 0 destroyed.
Outputs:
bigquery_dataset = {
"access" = [
{
"domain" = ""
"group_by_email" = ""
"role" = "OWNER"
"special_group" = "projectOwners"
"user_by_email" = ""
"view" = []
},
]
"creation_time" = 1605045037237
"dataset_id" = "foo"
"default_encryption_configuration" = []
"default_partition_expiration_ms" = 0
"default_table_expiration_ms" = 3600000
"delete_contents_on_destroy" = false
"description" = "some description"
"etag" = "pQEv+/zjV0cJk8J36Qyx0g=="
"friendly_name" = "foo"
"id" = "projects/ci-bigquery-76f4/datasets/foo"
"labels" = {
"billable" = "true"
"env" = "dev"
"owner" = "janesmith"
}
"last_modified_time" = 1605045037237
"location" = "US"
"project" = "ci-bigquery-76f4"
"self_link" = "https://bigquery.googleapis.com/bigquery/v2/projects/ci-bigquery-76f4/datasets/foo"
}
bigquery_tables = {
"bar" = {
"clustering" = []
"creation_time" = 1605045037864
"dataset_id" = "foo"
"description" = ""
"encryption_configuration" = []
"etag" = "V8yrtCB+HSQbTUvqgA5Y8g=="
"expiration_time" = 2524604400000
"external_data_configuration" = []
"friendly_name" = "bar"
"id" = "projects/ci-bigquery-76f4/datasets/foo/tables/bar"
"labels" = {
"billable" = "true"
"env" = "devops"
"owner" = "joedoe"
}
"last_modified_time" = 1605045037913
"location" = "US"
"materialized_view" = []
"num_bytes" = 0
"num_long_term_bytes" = 0
"num_rows" = 0
"project" = "ci-bigquery-76f4"
"range_partitioning" = []
"schema" = "[{\"description\":\"Full visitor ID\",\"mode\":\"NULLABLE\",\"name\":\"fullVisitorId\",\"type\":\"STRING\"},{\"description\":\"Visit number\",\"mode\":\"NULLABLE\",\"name\":\"visitNumber\",\"type\":\"INTEGER\"},{\"description\":\"Visit ID\",\"mode\":\"NULLABLE\",\"name\":\"visitId\",\"type\":\"INTEGER\"},{\"description\":\"Visit Start Time\",\"mode\":\"NULLABLE\",\"name\":\"visitStartTime\",\"type\":\"INTEGER\"},{\"description\":\"Full Date of Visit\",\"mode\":\"NULLABLE\",\"name\":\"fullDate\",\"type\":\"DATE\"}]"
"self_link" = "https://bigquery.googleapis.com/bigquery/v2/projects/ci-bigquery-76f4/datasets/foo/tables/bar"
"table_id" = "bar"
"time_partitioning" = []
"type" = "TABLE"
"view" = []
}
"foo" = {
"clustering" = [
"fullVisitorId",
"visitId",
]
"creation_time" = 1605045037974
"dataset_id" = "foo"
"description" = ""
"encryption_configuration" = []
"etag" = "Qx67/A84BaTpDXah2eaO5A=="
"expiration_time" = 1605048637974
"external_data_configuration" = []
"friendly_name" = "foo"
"id" = "projects/ci-bigquery-76f4/datasets/foo/tables/foo"
"labels" = {
"billable" = "true"
"env" = "dev"
"owner" = "joedoe"
}
"last_modified_time" = 1605045038028
"location" = "US"
"materialized_view" = []
"num_bytes" = 0
"num_long_term_bytes" = 0
"num_rows" = 0
"project" = "ci-bigquery-76f4"
"range_partitioning" = []
"schema" = "[{\"description\":\"Full visitor ID\",\"mode\":\"NULLABLE\",\"name\":\"fullVisitorId\",\"type\":\"STRING\"},{\"description\":\"Visit number\",\"mode\":\"NULLABLE\",\"name\":\"visitNumber\",\"type\":\"INTEGER\"},{\"description\":\"Visit ID\",\"mode\":\"NULLABLE\",\"name\":\"visitId\",\"type\":\"INTEGER\"},{\"description\":\"Visit Start Time\",\"mode\":\"NULLABLE\",\"name\":\"visitStartTime\",\"type\":\"INTEGER\"},{\"description\":\"Full Date of Visit\",\"mode\":\"NULLABLE\",\"name\":\"fullDate\",\"type\":\"DATE\"}]"
"self_link" = "https://bigquery.googleapis.com/bigquery/v2/projects/ci-bigquery-76f4/datasets/foo/tables/foo"
"table_id" = "foo"
"time_partitioning" = [
{
"expiration_ms" = 0
"field" = ""
"require_partition_filter" = false
"type" = "DAY"
},
]
"type" = "TABLE"
"view" = []
}
}
$$$$$$ Finished building the infrastructure based on the Terraform configuration.
$$$$$$ Reading the output variables from the Terraform state...
$$$$$$ Finished reading the output variables from the Terraform state.
$$$$$$ Parsing the Terraform output variables as JSON...
$$$$$$ Finished parsing the Terraform output variables as JSON.
$$$$$$ Writing the output variables to the Kitchen instance state...
$$$$$$ Finished writing the output varibales to the Kitchen instance state.
$$$$$$ Writing the input variables to the Kitchen instance state...
$$$$$$ Finished writing the input variables to the Kitchen instance state.
Finished converging <full-local> (0m13.48s).
-----> Test Kitchen is finished. (0m16.68s)
-----> Starting Test Kitchen (v2.7.2)
-----> Setting up <full-local>...
Finished setting up <full-local> (0m0.00s).
-----> Verifying <full-local>...
$$$$$$ Reading the Terraform input variables from the Kitchen instance state...
$$$$$$ Finished reading the Terraform input variables from the Kitchen instance state.
$$$$$$ Reading the Terraform output variables from the Kitchen instance state...
$$$$$$ Finished reading the Terraform output varibales from the Kitchen instance state.
$$$$$$ Verifying the systems...
$$$$$$ Verifying the 'system' system...
Failed to load profile local: Failed to load source for controls/big_query.rb: Bad response: {
"error": {
"code": 400,
"message": "Invalid project ID ''. Project IDs must contain 6-63 lowercase letters, digits, or dashes. Some project IDs also include domain name separated by a colon. IDs must start with a letter and may not end with a dash.",
"errors": [
{
"message": "Invalid project ID ''. Project IDs must contain 6-63 lowercase letters, digits, or dashes. Some project IDs also include domain name separated by a colon. IDs must start with a letter and may not end with a dash.",
"domain": "global",
"reason": "invalid"
}
],
"status": "INVALID_ARGUMENT"
}
}
Profile: local
Version: 0.1.0
Failure Message: Failed to load source for controls/big_query.rb: Bad response: {
"error": {
"code": 400,
"message": "Invalid project ID ''. Project IDs must contain 6-63 lowercase letters, digits, or dashes. Some project IDs also include domain name separated by a colon. IDs must start with a letter and may not end with a dash.",
"errors": [
{
"message": "Invalid project ID ''. Project IDs must contain 6-63 lowercase letters, digits, or dashes. Some project IDs also include domain name separated by a colon. IDs must start with a letter and may not end with a dash.",
"domain": "global",
"reason": "invalid"
}
],
"status": "INVALID_ARGUMENT"
}
}
Target: gcp://[email protected]
No tests executed.
Profile: Google Cloud Platform Resource Pack (inspec-gcp)
Version: 1.8.1
Target: gcp://[email protected]
No tests executed.
Test Summary: 0 successful, 0 failures, 0 skipped
>>>>>> Verifying the 'system' system failed:
Running InSpec failed:
Running InSpec failed due to a non-zero exit code of 1.
>>>>>> ------Exception-------
>>>>>> Class: Kitchen::ActionFailed
>>>>>> Message: 1 actions failed.
>>>>>> Verify failed on instance <full-local>. Please see .kitchen/logs/full-local.log for more details
>>>>>> ----------------------
>>>>>> Please see .kitchen/logs/kitchen.log for more details
>>>>>> Also try running `kitchen diagnose --all` for configuration
-----> Starting Test Kitchen (v2.7.2)
-----> Destroying <full-local>...
$$$$$$ Verifying the Terraform client version is in the supported interval of >= 0.11.4, < 0.14.0...
$$$$$$ Reading the Terraform client version...
Terraform v0.13.5
+ provider registry.terraform.io/hashicorp/external v2.0.0
+ provider registry.terraform.io/hashicorp/google v3.47.0
+ provider registry.terraform.io/hashicorp/null v3.0.0
+ provider registry.terraform.io/hashicorp/random v3.0.0
$$$$$$ Finished reading the Terraform client version.
$$$$$$ Finished verifying the Terraform client version.
$$$$$$ Initializing the Terraform working directory...
Initializing modules...
Initializing the backend...
Initializing provider plugins...
- Using previously-installed hashicorp/google v3.47.0
- Using previously-installed hashicorp/null v3.0.0
- Using previously-installed hashicorp/random v3.0.0
- Using previously-installed hashicorp/external v2.0.0
The following providers do not have any version constraints in configuration,
so the latest version was installed.
To prevent automatic upgrades to new major versions that may contain breaking
changes, we recommend adding version constraints in a required_providers block
in your configuration, with the constraint strings suggested below.
* hashicorp/external: version = "~> 2.0.0"
* hashicorp/null: version = "~> 3.0.0"
* hashicorp/random: version = "~> 3.0.0"
Terraform has been successfully initialized!
$$$$$$ Finished initializing the Terraform working directory.
$$$$$$ Selecting the kitchen-terraform-full-local Terraform workspace...
$$$$$$ Finished selecting the kitchen-terraform-full-local Terraform workspace.
$$$$$$ Destroying the Terraform-managed infrastructure...
module.example.module.bigquery.google_bigquery_dataset.main: Refreshing state... [id=projects/ci-bigquery-76f4/datasets/foo]
module.example.module.bigquery.google_bigquery_table.main["foo"]: Refreshing state... [id=projects/ci-bigquery-76f4/datasets/foo/tables/foo]
module.example.module.bigquery.google_bigquery_table.main["bar"]: Refreshing state... [id=projects/ci-bigquery-76f4/datasets/foo/tables/bar]
module.example.module.bigquery.google_bigquery_table.main["bar"]: Destroying... [id=projects/ci-bigquery-76f4/datasets/foo/tables/bar]
module.example.module.bigquery.google_bigquery_table.main["foo"]: Destroying... [id=projects/ci-bigquery-76f4/datasets/foo/tables/foo]
module.example.module.bigquery.google_bigquery_table.main["foo"]: Destruction complete after 0s
module.example.module.bigquery.google_bigquery_table.main["bar"]: Destruction complete after 0s
module.example.module.bigquery.google_bigquery_dataset.main: Destroying... [id=projects/ci-bigquery-76f4/datasets/foo]
module.example.module.bigquery.google_bigquery_dataset.main: Destruction complete after 1s
Destroy complete! Resources: 3 destroyed.
$$$$$$ Finished destroying the Terraform-managed infrastructure.
$$$$$$ Selecting the default Terraform workspace...
Switched to workspace "default".
$$$$$$ Finished selecting the default Terraform workspace.
$$$$$$ Deleting the kitchen-terraform-full-local Terraform workspace...
Deleted workspace "kitchen-terraform-full-local"!
$$$$$$ Finished deleting the kitchen-terraform-full-local Terraform workspace.
Finished destroying <full-local> (0m12.25s).
-----> Test Kitchen is finished. (0m15.48s)
Makefile:61: recipe for target 'docker_test_integration' failed
make: *** [docker_test_integration] Error 20
Minor issue but in few places wheree is misspelled.
Google supports materialized views in BigQuery and we'd like to be able to define them via terraform.
https://cloud.google.com/bigquery/docs/materialized-views-intro
From what we can tell, the existing table/view resources do not support them yet.
I'm not sure if this is a bug report or a feature request.
If a big query table is created with a column that has the mode left unspecified and then later the mode is changed to "NULLABLE" terraform plan will indicate an update is required:
~ schema = jsonencode(
~ [
~ {
+ mode = "NULLABLE"
name = "AccountID"
type = "NUMERIC"
},
<ommitted for brevity>
]
)
Upon running terraform apply Terraform will indicate that the "modification was completed successfully" when no change was actually made. Running terraform plan again will result in the same plan as before. The converse is also true, if the mode is initially "NULLABLE" and then later changed to unspecified the provider will indicate that the mode is going to be removed.
I understand why no change is made but I think, given that mode defaults to "NULLABLE" when no mode is specified, the provider should treat these as the same and not indicate that an update is required.
Add a upgrade guide for users migrating from TF11 to TF12
Current Issue: Trying to pair a table creation with "list of maps" fails when a lookup is performed on the table labels
v0.2.0 May lack support for multiple tables with distinct labels as lookups only work with flat maps and there will be more rich functionality available in tf 12 to handle this need.
Hi,
For this module, I couldn't find support for expiration_time for a view. Default expiration is considered even after passing 'expiration_time'. I don't think custom expiration_time is considered as part of input, its not throwing error but is not considered.
Will it be possible to implement ?
Thank you.
Hi there
I am using the provider to:
I followed this documentation.
The final step fails with this error:
Error applying IAM policy for KMS CryptoKey "projects/[PROJECT_ID]/locations/europe/keyRings/[KEYRING_NAME]/cryptoKeys/[CRYPTOKEY_NAME]": Error setting IAM policy for KMS CryptoKey "projects/[PROJECT_ID]/locations/europe/keyRings/[KEYRING_NAME]/cryptoKeys/[CRYPTOKEY_NAME]": googleapi: Error 400: Service account bq-[PROJECT_NUMBER]@bigquery-encryption.iam.gserviceaccount.com does not exist., badRequest
The issue also arises if I try to grant the Encryptor/Decryptor role in the IAM pane through the GCP Console GUI.
Strangely, the issue is not permanent - it seems to resolve within approximately 24 hours. All of a sudden, the service account exists and is available for granting IAM roles. Granting the required role either through the Console or by re-queueing the Terraform Plan and Apply is successful.
I can't see any pattern regarding what resolves the issue. Initially, we suspected that the account may auto-create after a first dataset is created. So we now have a pre-step that makes a dummy dataset, waits 30 mins and then completes the steps above. This still fails.
Code below:
// Get the project number for the environment.
data "google_project" "project_number" {}
// Create a dummy dataset to prime the BigQuery service account. NOTE: This dataset is not actually to be used.
resource "google_bigquery_dataset" "dataset_dummy" {
dataset_id = "${var.environment_name}_dataset_dummy"
location = var.bigquery_dataset_location
default_table_expiration_ms = var.bigquery_dataset_default_table_expiration_ms
description = "DO NOT USE THIS DATASET. Do not store any data here. It is created simply to prime the BigQuery service account."
}
// Sleep after creating the dataset. This allows the BigQuery service account to be created, enabling the use of CMEKs.
resource "time_sleep" "bigquery_wait" {
depends_on = [google_bigquery_dataset.dataset_dummy]
create_duration = "30m"
}
// Bind the BigQuery service account to the BigQuery CMEK, once the delay has finished.
resource "google_kms_crypto_key_iam_binding" "bigquery_dataset_cmek_iam" {
crypto_key_id = var.bigquery_dataset_cmek_id
role = "roles/cloudkms.cryptoKeyEncrypterDecrypter"
members = [
"serviceAccount:bq-${data.google_project.project_number.number}@bigquery-encryption.iam.gserviceaccount.com",
]
depends_on = [time_sleep.bigquery_wait]
}
// Sleep again after conducting the IAM binding, giving time for it to propagate.
resource "time_sleep" "bigquery_cmek_iam_wait" {
depends_on = [google_kms_crypto_key_iam_binding.bigquery_dataset_cmek_iam]
create_duration = "5m"
}
// Create a default dataset to house all tables in this project.
resource "google_bigquery_dataset" "dataset_1" {
dataset_id = "${var.environment_name}_dataset_1"
location = var.bigquery_dataset_location
default_table_expiration_ms = var.bigquery_dataset_default_table_expiration_ms
// Encrypt the dataset with a CMEK. (https://cloud.google.com/bigquery/docs/customer-managed-encryption)
default_encryption_configuration {
kms_key_name = var.bigquery_dataset_cmek_id
}
// Wait for the CMEK IAM binding to propagate.
depends_on = [time_sleep.bigquery_cmek_iam_wait]
}
Environment details:
Please help! We need to understand the logic behind what spawns the BigQuery service account, so that we can add appropriate waits/retries into our build specification.
Thank you :)
To help users leveraging beta and google provider ~2.0.0 the module and examples should be modified to accommodate for this Upgrade process
If the tables are in a different order in the schema in terraform, terraform does an update that does nothing.
Every time you run terraform apply it tries to do a change to the table.
When columns are added to a bigquery table they are appended to the end of the table.
Customers asked for 2 options:
Warning: External references from destroy provisioners are deprecated
on .terraform/modules/dataset.udfs.bq_check_protocol/main.tf line 179, in resource "null_resource" "run_command":
179: command = <<-EOT
180: PATH=${local.gcloud_bin_abs_path}:$PATH
181: ${var.destroy_cmd_entrypoint} ${var.destroy_cmd_body}
182: EOT
Destroy-time provisioners and their connection configurations may only
reference attributes of the related resource, via 'self', 'count.index', or
'each.key'.
References to other resources during the destroy phase can cause dependency
cycles and interact poorly with create_before_destroy.
(and 31 more similar warnings elsewhere)
We should update the testing to use Cloud Build and a new Service Account per run, as documented here.
Hi there!
In the Terraform google_bigquery_table ressource, schema expects a JSON string, however this module forces schema to be a file path because file() is used:
terraform-google-bigquery/main.tf
Line 60 in ebe17b8
I think this takes away some of the original flexibility of google_bigquery_table. I prefer using this well designed module over doing everything BQ separately, however in my current case I already had all the schemas in a map.
Just a nice to have! 👍
Make them actually demonstrate module usage.
Notes: #3 (review)
Hi,
I am new to BQ w/terraform.
I was planning on following this Hashicorp document to use terraform with BQ:
https://www.terraform.io/docs/providers/google/r/bigquery_table.html
But I came across this module.
Can someone comment on what direction a new project should take in this regard? The provider in Hashicorp documentation, or this module? Which may have better long term supportability?
Based on my reading of the API and the module, it is currently impossible to create a BigQuery dataset or table without setting default expiration IDs. Requiring data to expire after a certain amount of time should not be a requirement to use this module.
If we need to choose between specifying a default timeout and not, I vote not.
Implement the following native kitchen inspec resources into the test suite.
google_bigquery_table
google_bigquery_dataset
So I was looking for something that might allow me to define a UDF inside a Google Cloud BigQuery dataset, and wasn't able to find anything here. It seems like the Go client library allows for creation of these by creation of a Routine, defined here.
It might be worthwhile to consider adding this functionality to the module, considering they can now be defined in a persistent manner (see this)
https://cloud.google.com/blog/products/data-analytics/introducing-the-bigquery-terraform-module
The variable.tf code isn't visible (tf has a hyperlink)
The UDF module in this repo hard codes some specific persistent UDFs that one particular customer cared about. I don't think CFT modules is the right place for us to distribute UDFs as all of these (except check protocol) have already been donated to https://github.com/GoogleCloudPlatform/bigquery-utils/tree/master/udfs/community.
I'm not sure why the source for the UDFs is also duplicated in a bash script (which appears unused)
cc: @bharathkkb @umairidris @milesmatthias
IHAC that would be keen to consume a general BigQuery UDF module rather than one that hard codes these particular UDFs.
Would it be ok if I open a PR that overwrites this module? we can take the existing UDF definitions and throw them in an example for this submodule.
@WillBeebe can you please ask our favorite ad exchange if they depend on this UDF submodule directly from this OSS repo and help them migrate if so?
BigQuery supports drop column on DDL without the need to recreate the table. Do you have a timeline for when this is suppose to be supported on Terraform. We currently use 'hashicorp/google-beta'. Tnx
BigQuery now supports external BigTable-backed tables - this module has support for the other external types but unfortunately not these and it would be very useful to support them.
Documentation: https://cloud.google.com/bigquery/external-data-bigtable
The schema is configured in a JSON file like the other formats.
It seems for datasets the expected primitive role is "WRITER" instead of "EDITOR". I was getting an error after the conversion of "roles/bigquery.dataEditor" to "EDITOR" using the latest code.
Workaround: I just switched it to "WRITER" instead of "roles/bigquery.dataEditor" as a workaround.
Documentation: https://cloud.google.com/bigquery/docs/access-control-primitive-roles#dataset-primitive-roles
Issue came from this PR: #62
We should switch to for_each for obvious reasons.
On newer versions of the provider, you must explicitly set deletion_protection=false in order to destroy an instance.
google_bigquery_table resources now cannot be destroyed unless deletion_protection = false is set in state for the resource.
Current version module hasn't had attribute deletion_protection
The module should be able to create/manage multiple tables for the same data set.
BigQuery is one of the major products that healthcare customers come to GCP for. Having a clean, reliable and secure module is very important from our perspective.
With the many changes I recently proposed I hope we can get them merged in and make a major release which we can use with customers going forward. Given there might be some maintenance overhead from this change and potential future changes I am happy to co-own this module representing HCLS.
add file function to query parameter to accept .sql file as a input
I'd like to create bigquery tables via terraform.
The tables I plan to create are partitionned tables (i.e. requires a schema by default).
This schema will potentially change in the future if I need to add some columns to my table.
The data stored in the table isnt easy to reload.
Though it looks like updating the schema will completely recreate the table as mentionned here.
So does this "any changes on the configured value will force the table to be recreated." mean I wont be able to update the schema of the table on the fly while the data isnt dropped?
Thank you for the help and clarification
Makefile, test/ci_integration.sh, and related files based on https://github.com/terraform-google-modules/terraform-google-module-template.Hi! I'm using Terraform to define a few datasets in BQ and sometimes actions other than a TF create tables. How does TF define a new table vs an existing one in the module? I don't see anything along the lines of a flag (other than creation time, which we aren't using).
Please add sample code to use this module in the main README.md
Hello,
I tried basic_bq/main.tf example (link), but adding source and version see below. (I also have sample_bq_schema.json as per this link.
module "bigquery" {
source = "terraform-google-modules/bigquery/google"
version = "~> 4.4"
dataset_id = "foo"
dataset_name = "foo"
description = "some description"
project_id = "test-project-id"
location = "US"
delete_contents_on_destroy = null
tables = [
{
table_id = "bar",
schema = file("sample_bq_schema.json"),
time_partitioning = null,
range_partitioning = null,
expiration_time = 2524604400000, # 2050/01/01
clustering = [],
labels = {
env = "devops"
billable = "true"
owner = "joedoe"
},
}
]
dataset_labels = {
env = "dev"
billable = "true"
owner = "janesmith"
}
}
but I got the following error.
------------------------------------------------------------------------
Error: Error in function call
on .terraform/modules/bigquery/main.tf line 68, in resource "google_bigquery_table" "main":
68: schema = file(each.value["schema"])
|----------------
| each.value["schema"] is "[\n {\n \"description\": \"Full visitor ID\",\n \"mode\": \"NULLABLE\",\n \"name\": \"fullVisitorId\",\n \"type\": \"STRING\"\n },\n {\n \"description\": \"Visit number\",\n \"mode\": \"NULLABLE\",\n \"name\": \"visitNumber\",\n \"type\": \"INTEGER\"\n },\n {\n \"description\": \"Visit ID\",\n \"mode\": \"NULLABLE\",\n \"name\": \"visitId\",\n \"type\": \"INTEGER\"\n },\n {\n \"description\": \"Visit Start Time\",\n \"mode\": \"NULLABLE\",\n \"name\": \"visitStartTime\",\n \"type\": \"INTEGER\"\n },\n {\n \"description\": \"Full Date of Visit\",\n \"mode\": \"NULLABLE\",\n \"name\": \"fullDate\",\n \"type\": \"DATE\"\n }\n]\n"
Call to function "file" failed: failed to read [
{
"description": "Full visitor ID",
"mode": "NULLABLE",
"name": "fullVisitorId",
"type": "STRING"
},
{
"description": "Visit number",
"mode": "NULLABLE",
"name": "visitNumber",
"type": "INTEGER"
},
{
"description": "Visit ID",
"mode": "NULLABLE",
"name": "visitId",
"type": "INTEGER"
},
{
"description": "Visit Start Time",
"mode": "NULLABLE",
"name": "visitStartTime",
"type": "INTEGER"
},
{
"description": "Full Date of Visit",
"mode": "NULLABLE",
"name": "fullDate",
"type": "DATE"
}
]
.
Did I miss something?
Thank you,
Laurentius Purba
Error: Unsupported attribute
on outputs.tf line 33, in output "table_id":
33: value = google_bigquery_table.main.*.id
This object does not have an attribute named "id".
Error: Unsupported attribute
on outputs.tf line 38, in output "table_name":
38: value = google_bigquery_table.main.*.friendly_name
This object does not have an attribute named "friendly_name".
Error: Unsupported attribute
on outputs.tf line 48, in output "table_labels":
48: value = google_bigquery_table.main.*.labels
This object does not have an attribute named "labels".
All users that need to consume the data set need at minimum dataset viewer. Would best to have an array of users and be able to access.
I am trying to create a bigquery job and insert data from a sql cloud instance database table. For data ingestion, I have setup external connection with cloud SQL in GCP and can create Schedule Query using MERGE INTO(SQL query), using bigquery UI. As a result of creating this query I am able to insert data into BQ from sql instance.
For doing the same task using terraform I have setup dataset, table and external connection. But the blocker is google_bigquery_job in terraform. Even though the dataset is available in BQ, on running the script I am receiving following error:
Error: Error creating Job: googleapi: Error 404: Not found: Dataset (Here Dataset ID), notFound
Related Script is as follows:
` resource "google_bigquery_job" "job" {
provider=google
job_id = "job_query_testing"
labels = {
"example-label" ="example-value"
}
query {
query = <<EOF SQL query here EOF
use_legacy_sql = false
write_disposition="WRITE_APPEND"
destination_table {
project_id = var.project_id
dataset_id = google_bigquery_dataset.main.dataset_id
table_id = google_bigquery_table.main.table_id
}
allow_large_results = true
script_options {
key_result_statement = "LAST"
}
}
}
`
Currently terraform module requires schema to be defined for table creation. According to [1] and the tf native resource [2] , a schema can be optional. What should occur is:
Schema [Optional] The schema for the data. Schema is required for CSV and JSON formats if autodetect is not on. Schema is disallowed for Google Cloud Bigtable, Cloud Datastore backups, Avro, ORC and Parquet formats.
Ask: Allow schema to be optional and support the ability to enable autodetect.
[1] https://cloud.google.com/bigquery/docs/reference/rest/v2/tables#TableSchema
[2] https://www.terraform.io/docs/providers/google/r/bigquery_table.html
Hey,
I like the library, this could potentially replace a custom liquibase change management system I was previously using with bigquery.
I was wondering if there are any plans to support other bigquery primitives, such as views, functions, or procedures? I couldn't find anything in previous issues
Thanks
The access resource for a bigquery dataset is an ordered list. When comparing to the state file, the diff displays every item being removed and re-added, in addition to any actual additions/removals.
Prior to 0.12 terraform, there were workarounds for this (I forked the landscape repo and modified it to order these lists prior to comparing them). However, landscape doesn't support 0.12 terraform, and I imagine it won't any time soon.
Would it work to change the data type of the access resource from a list to a set? I have not dabbled much in configuring terraform providers.
Have others encountered this issue? If so, how do you deal with it?
variable "list_of_objects_for_granting_access" {
description = "The list of users with explicit access to the my fav dataset"
type = list(object({
email = string,
email_type = string,
role = string}))
default = []
}
resource "google_bigquery_dataset" "my_fav_dataset" {
dataset_id = "my_fav_dataset"
project = google_project.fav_module_name.project_id
friendly_name = "fav dataset"
description = "where all the fav tables go"
dynamic "access" {
for_each = flatten([var.list_of_objects_for_granting_access])
content {
role = access.value["role"]
group_by_email = access.value["email_type"] != "user" ? access.value["email"] : null
user_by_email = access.value["email_type"] != "group" ? access.value["email"] : null
}
}
access {
role = "OWNER"
group_by_email = "[email protected]"
}
access {
role = "READER"
user_by_email = "[email protected]"
}
}
tf plan:
# module.stage_fav_1.google_bigquery_dataset.fav_dataset will be updated in-place
~ resource "google_bigquery_dataset" "fav_dataset" {
creation_time = 1555962987200
dataset_id = "fav_dataset"
default_partition_expiration_ms = 0
default_table_expiration_ms = 0
delete_contents_on_destroy = false
description = "Dataset to house fav Data"
etag = ""
friendly_name = "fave data"
id = "fav-1-stage:fav_dataset"
labels = {}
last_modified_time = 1581440087969
location = "US"
project = "fav-1-stage"
self_link = ""
- access {
- view {
- dataset_id = "beta" -> null
- project_id = "fav2-reporting" -> null
- table_id = "deliveryinfo_zones" -> null
}
}
- access {
- view {
- dataset_id = "fav_dataset" -> null
- project_id = "fav-stage" -> null
- table_id = "auth_user" -> null
}
}
- access {
- view {
- dataset_id = "fav_dataset" -> null
- project_id = "fav-stage" -> null
- table_id = "core_contact" -> null
}
}
- access {
- view {
- dataset_id = "fav_dataset" -> null
- project_id = "fav-stage" -> null
- table_id = "core_contactaddress" -> null
}
}
- access {
- view {
- dataset_id = "fav_dataset" -> null
- project_id = "fav-stage" -> null
- table_id = "core_applicant" -> null
}
}
- access {
- view {
- dataset_id = "fav_dataset" -> null
- project_id = "fav-stage" -> null
- table_id = "core_customer" -> null
}
}
- access {
- view {
- dataset_id = "fav_dataset" -> null
- project_id = "fav-stage" -> null
- table_id = "core_customerguest" -> null
}
}
- access {
- view {
- dataset_id = "fav_dataset" -> null
- project_id = "fav-stage" -> null
- table_id = "core_info" -> null
}
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "READER" -> null
- user_by_email = "[email protected]" -> null
}
- access {
- role = "WRITER" -> null
- user_by_email = "[email protected]" -> null
}
+ access {
+ role = "WRITER"
+ user_by_email = "[email protected]"
}
- access {
- group_by_email = "[email protected]" -> null
- role = "READER" -> null
}
- access {
- group_by_email = "[email protected]" -> null
- role = "OWNER" -> null
}
- access {
- group_by_email = "[email protected]" -> null
- role = "READER" -> null
}
- access {
- group_by_email = "an-email-group.com" -> null
- role = "READER" -> null
}
- access {
- group_by_email = "[email protected]" -> null
- role = "READER" -> null
}
- access {
- group_by_email = "[email protected]" -> null
- role = "READER" -> null
}
+ access {
+ group_by_email = "[email protected]"
+ role = "READER"
}
+ access {
+ group_by_email = "[email protected]"
+ role = "OWNER"
}
+ access {
+ group_by_email = "[email protected]"
+ role = "READER"
}
+ access {
+ group_by_email = "[email protected]"
+ role = "READER"
}
+ access {
+ group_by_email = "[email protected]"
+ role = "READER"
}
+ access {
+ group_by_email = "[email protected]"
+ role = "READER"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ role = "READER"
+ user_by_email = "[email protected]@fav2.com"
}
+ access {
+ view {
+ dataset_id = "beta"
+ project_id = "fav2-reporting"
+ table_id = "zones"
}
}
+ access {
+ view {
+ dataset_id = "fav_dataset"
+ project_id = "fav-stage"
+ table_id = "auth_user"
}
}
+ access {
+ view {
+ dataset_id = "fav_dataset"
+ project_id = "fav-stage"
+ table_id = "core_contact"
}
}
+ access {
+ view {
+ dataset_id = "fav_dataset"
+ project_id = "fav-stage"
+ table_id = "core_address"
}
}
+ access {
+ view {
+ dataset_id = "fav_dataset"
+ project_id = "fav-stage"
+ table_id = "core_applicant"
}
}
+ access {
+ view {
+ dataset_id = "fav_dataset"
+ project_id = "fav-stage"
+ table_id = "core_customer"
}
}
+ access {
+ view {
+ dataset_id = "fav_dataset"
+ project_id = "fav-stage"
+ table_id = "core_guest"
}
}
+ access {
+ view {
+ dataset_id = "fav_dataset"
+ project_id = "fav-stage"
+ table_id = "core_info"
}
}
This is using 0.12.20 terraform and 3.8.0 google tf provider.
Any dataset table list modification (e.g. new table, modification, or removal operation) triggers all exiting table recreation.
terraform 0.14
module 4.x/5
encryption defined (with dataset encryption_key).
table A and some encryption keytable B with no other changes to tables A / dataset.Plan output fragment
Terraform will perform the following actions:
# module.project_bigquery["my_bq_dataset"].google_bigquery_table.main["a"] must be replaced
-/+ resource "google_bigquery_table" "main" {
~ creation_time = 1620270233757 -> (known after apply)
~ etag = "45iAIgM/cjwITp8LOlZQlA==" -> (known after apply)
~ id = "projects/X/datasets/my_bq_dataset/tables/a" -> (known after apply)
~ last_modified_time = 1620270233818 -> (known after apply)
~ location = "asia-southeast2" -> (known after apply)
~ num_bytes = 0 -> (known after apply)
~ num_long_term_bytes = 0 -> (known after apply)
~ num_rows = 0 -> (known after apply)
~ self_link = "https://bigquery.googleapis.com/bigquery/v2/projects/X/datasets/my_bq_dataset/tables/a" -> (known after apply)
~ type = "TABLE" -> (known after apply)
# (7 unchanged attributes hidden)
- encryption_configuration { # forces replacement
- kms_key_name = "projects/Y/locations/asia-southeast2/keyRings/x-az1/cryptoKeys/default" -> null
}
~ time_partitioning {
~ expiration_ms = 0 -> (known after apply)
# (2 unchanged attributes hidden)
}
}
# module.project_bigquery["my_bq_dataset"].google_bigquery_table.main["b"] will be created
+ resource "google_bigquery_table" "main" {
+ clustering = []
+ creation_time = (known after apply)
+ dataset_id = "my_bq_dataset"
+ etag = (known after apply)
+ expiration_time = 2524604400000
+ friendly_name = "b"
+ id = (known after apply)
+ last_modified_time = (known after apply)
+ location = (known after apply)
+ num_bytes = (known after apply)
+ num_long_term_bytes = (known after apply)
+ num_rows = (known after apply)
+ project = "X"
+ schema = jsonencode(
[
+ {
+ description = "CLNTPFX"
+ mode = "REQUIRED"
+ name = "CLNTPFX"
+ type = "STRING"
},
]
)
+ self_link = (known after apply)
+ table_id = "b"
+ type = (known after apply)
+ time_partitioning {
+ expiration_ms = (known after apply)
+ require_partition_filter = false
+ type = "DAY"
}
}
Plan: 2 to add, 0 to change, 1 to destroy.
Table B will be created with no impact to Table A
The encryption attribute is defined by design as a dataset and manages (enforce) encryption for all tables. Table terraform resource itself has encryption attribute. Initial table creation inherits encryption value from a dataset but any next terraform action will set table encryption to "null", encryption key change forces table recreation and data loss.
Currently, to use the bigquery module users need to set the google and google-beta provider version to ~>2.15. That would nice to upgrade it to the 3.0.0 version at least. Thanks!
modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.v0.12.19+
Terraform modules can be referenced in the config files using following syntax:
module "<module-name>" {
source = "<source>"
version = "<version-constraint>"
// module attributes
}Reference: https://www.terraform.io/docs/configuration/modules.html
Both source and version attributes have flexibility in how they are expressed
source = "terraform-google-modules/bigquery/google"
version = "~> 4.0"This means it is a "terraform registry" module which supports a specific discovery/lookup method to find the location of the module to download; the actual download can be from anywhere though typically it’s a github repo. The version attribute represents a constraint telling terraform to restrict versions of the module to at least version 4.0. (The version attribute only works with "terraform registry" modules that use this syntax.) The download happens via HTTP(s) GET request.
source = "github.com/terraform-google-modules/terraform-google-bigquery?ref=v4.2.1"
source = "git::https://github.com/terraform-google-modules/terraform-google-bigquery?ref=v4.2.1"
source = "github.com/terraform-google-modules/terraform-google-bigquery" Using method #2 the version attribute in the module stanza is not supported. If a specific version is needed it must be specified in the ref= query param in the url. (Leaving it off means you will get the latest version - currently 4.2.1).
All the above examples are valid and all result in the same version of the google module that gets downloaded; but in all of these cases the download is via "git" protocol (either https or ssh clone method).
There a problem using method #1 of referencing modules versus method #2 on certain versions of Terraform.
With Terraform v0.12.19 thru v0.12.26 (latest version)
source = "terraform-google-modules/bigquery/google", the terraform discovery process results in a download location of:https://github.com/terraform-google-modules/terraform-google-bigquery/archive/v4.2.1.tar.gz?archive=tar.gzhttps://codeload.github.com/terraform-google-modules/terraform-google-bigquery/tar.gz/v4.2.1With Terraform versions up to and including v0.12.18 there is no issue with either method #1 or #2; In these versions, the terraform discovery process results in the same "git" download location as when using the method #2.
Since the end result of the terraform registry lookup method (in recent terraform versions) results in a significantly different result for the location (note the paths to the repo are also different; even though the resulting files are same one is an archive reference) and because the redirect points to a site with invalid cert, we believe this to be a bug with how these modules are configured in the terraform registry and/or a bug in the newer terraform versions.
We will use method #2 to reference the modules for now since that seems to work.
Hi,
It would be great if we could create BigQuery tables with setting clustering columns.
Documentation: https://cloud.google.com/bigquery/docs/creating-clustered-tables
Affected Resource(s)
google_bigquery_table
Hello @morgante
Wondering if there are plans to support the "view" block in the v4.0.0 access variable?
Currently only part of the access block is used: https://github.com/terraform-google-modules/terraform-google-bigquery/blob/master/main.tf#L31-L41
I'm referring to this view block here on the dataset resource: https://www.terraform.io/docs/providers/google/r/bigquery_dataset.html#view
This would be great as it would allow us to create authorized views easily.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.