Giter Site home page Giter Site logo

terraform-ibm-modules / terraform-ibm-cos Goto Github PK

View Code? Open in Web Editor NEW
7.0 20.0 23.0 1.4 MB

Configures an IBM Cloud Object Storage instance and bucket

License: Apache License 2.0

HCL 83.87% Go 11.76% Shell 4.37%
terraform-module ibm-cloud cloud-object-storage cos-bucket terraform core-team graduated supported bucket cos

terraform-ibm-cos's Issues

Add support for One-rate plan

Description

The variable cos_plan only allows standard and lite plans however it seems there is a third plan - "One Rate":

The One-rate plan offers large enterprises and ISVs (with multiple divisions or end-users) a predictable TCO based on one flat rate. The flat rate has built-in allowances for outbound bandwidth and operational requests. These allowances depend on the amount of storage capacity aggregated across multiple instances within a One Rate pricing region. There is no data retrieval charge.

New or affected modules

https://github.com/terraform-ibm-modules/terraform-ibm-cos

By submitting this issue, you agree to follow our Code of Conduct

Timing issue with auth policy when creating multiple buckets with buckets submodule

When you are using the buckets or fscloud submodule, and you are creating more than 1 bucket with KMS encryption enabled, and have skip_iam_authorization_policy set to false for 1 bucket and set to true for the other, there is a timing issue where 1 bucket config will start to create the KMS auth policy first, while the other will start to create the bucket first, but bucket creation will fail because the auth policy may not yet be created.

We found this issue in terraform-ibm-modules/terraform-ibm-observability-da#8

Ideally we need to find a way where if any bucket config is set to create an auth policy, NO buckets should get created until its created. This logic would have to live in the buckets submodule I guess.

We probably should also fast fail if we detect that multiple buckets have skip_iam_authorization_policy set to false since its going to try and create duplicate policies and fail anyway.

CBR rules are not applied to bucket and instance (incorrect ids passed in the rules)

Scenario to reproduce the issue:

  1. Execute the complete examples at https://github.com/terraform-ibm-modules/terraform-ibm-cos/tree/main/examples/complete - this example creates 3 CBR rules (one for instance, and one per bucket) that are scoped to allow access only from a VPC (created in the example as well)
  2. Change the CBR enforcement mode to 'enabled' (example set it to reporting by default)
  3. Navigate to the COS instance from your local browser (so that you're not in the VPC..) - from the IBM cloud console for instance.

Result: Notice you can see the instance and buckets
Expected: Access denied message when accessing the instance as requests are not coming from within VPC (but the internet in this case). As below (displayed when fixing the CBR rule manually)

image

Issue is that the module pass the instance id as input, instead of the guid. https://github.com/terraform-ibm-modules/terraform-ibm-cos/blob/main/main.tf#L236

Affected modules

  • terraform-ibm-cos
  • Seeing the same problem across all modules using the same CBR code - so it would be good to do a sweep around

See whether we could expend the automated test suite as well to catch this type of issue - eg: trying to list the cos bucket in the instance as part of the test executing the complete example would catch this type of problem.

Variable descriptions in COS fscloud submodule need to be updated

The descriptions of the below variable in the FSCloud submodule seem to indicated they are required if 'create_cos_bucket' is true, however there is no create_cos_bucket variable in the FSCloud submodule. They are also not required variables at all (which I think should be correct - the user can optionally opt in).

variable "activity_tracker_crn" {
  type        = string
  description = "Activity tracker crn for COS bucket. Only required if 'create_cos_bucket' is true."
  default     = null
}

variable "sysdig_crn" {
  type        = string
  description = "Sysdig Monitoring crn for COS bucket. Only required if 'create_cos_bucket' is true."
  default     = null
}

Expose COS resource key details as outputs

The COS module supports creating a resource key, however it does not output it in its outputs.
We should add 3 new outputs to output the name, ID and value of the key.

The problem we will face is when we output the key value as a sensitive value, the buckets module complains because it has the following output:

output "buckets" {
  description = "Map of buckets created in the Cloud Object Storage Instance"
  value       = module.buckets
}

The output will now contain a sensitive value and hence terraform says it should be marked as a sensitive, which is not what we want. We should update the logic here so it only outputs bucket info. There is no need to include the resource key output in this output.

Ensure we also add the outputs to the fscloud submodule.

Retention and cross-region error message

Description

module "cos" {
  resource_group_id      = module.resource_group.resource_group_id
  region                 = var.region
  cos_instance_name      = "${var.prefix}-cos"
  cos_tags               = var.resource_tags
  bucket_name            = "${var.prefix}-bucket"
  retention_enabled      = true 
}

This errors out with the following message.
validate_cross_region_retention = var.cross_region_location != "us" && var.retention_enabled ? tobool("Retention is currently only supported in the US location for cross region buckets.") : true

The issue is that at no point the call is retrying to create a cross region bucket.

Update COS bucket DAs to support to creating KMS key in external account

Changes should go into solutions/secure-cross-regional-bucket and solutions/secure-regional-bucket only..

  • Add new optional variable ibmcloud_kms_api_key. This should be used in a new provider block like so:

    provider "ibm" {
  alias            = "kms"
  ibmcloud_api_key = var.ibmcloud_kms_api_key != null ? var.ibmcloud_kms_api_key : var.ibmcloud_api_key
  region           = local.kms_region # this value should be parsed from the existing KMS CRN 
}

  • The kms module block should use the kms provider alias (its already set up like this in the code actually).

  • Support creating cross account s2s auth policy (in KMS account):

    • if value for ibmcloud_kms_api_key is passed, and skip_iam_authorization_policy is set to false, then create a cross account s2s auth policy in the KMS account to allow the COS bucket reader access to the KMS instance GUID in the KMS account.
    • Ensure that if doing cross account auth policy, the skip_iam_authorization_policy value thats passed to the COS module itself is set to true since we will create the cross account one in the DA itself.

  • Review all of the variable descriptions and readme markdowns to ensure its clear that it supports KMS in a different account using the ibmcloud_kms_api_key variable.

buckets submodule is missing the ability to set the endpoint type to use

The root level module has a variable called management_endpoint_type_for_bucket which can be set to "public", "private", or "direct", however there doesn't seem to be any way to set this when creating a bucket using the buckets submodule. That means all buckets get created with "public" since this is the default value in the root level module

'encyption_enabled' field name is misleading

Minor issue but could lead to confusion.

encryption_enabled controls whether kms encryption is enable (key protect / hpcs). If set to false, cos bucket is still encrypted but with ibm managed keys.

COS module needs updates around monitoring and activity tracking for buckets

Some changes are coming in IBM-Cloud/terraform-provider-ibm#5277 which will require some refactoring of the module...

Current code works like this:

  • When values are passed for activity_tracker_crn and sysdig_crn, this is what determines if activity tracking or monitoring should be enabled for buckets.

Refactoring considerations:

  • activity_tracker_crn and metrics_monitoring_crn are now becoming optional by the provider. If they are not specified, it will default to the default AT or Sysdig instances in the account. So with this in mind, we should probably create new variables enable_metrics_monitoring and enable_activity_tracking which default to true (see how upgrade test handles this).
  • management_events is also coming in new provider drop for AT, so that should be set to true as well.
  • take this opportunely to rename sysdig_crn to monitoring_crn

NB: modules/fscloud, modules/buckets, solutions/secure-cross-regional-bucket, solutions/secure-regional-bucket will all need updates based on the refactor.

Add output var for COS instance CRN

Description

Some downstream service DA's (eg. Observability) require the CRN of the COS service instance. Only guid and id are output from the current DA. This feature request is to add CRN to output values of the COS DA to make it easier to wire up downstream DA's.

New or affected modules

terraform-ibm-cos

By submitting this issue, you agree to follow our Code of Conduct

Allow naming of cos instance created by terraform-ibm-cos

Description

Most of the time we want to create a Cloud Object Storage instance with a predetermined name that adheres to a projects naming conventions.

The current behaviour when creating a COS instance is to use the name environment-name-cos.

New or affected modules

https://github.com/terraform-ibm-modules/terraform-ibm-cos

Other considerations

Existing consumers of this module that specify both create-cos-instance=true and cos-instance-name may see a change in behaviour. The current behaviour is to silently ignore the cos-instance-name provided. The new behaviour will be to change the name of the COS instance, which may result in an attempt to delete and re-create the instance.

Programmatically determine the `kms_region` from the CRN in COS DA

To be consistent with other DAs, both the COS buckets DAs should ask for:

  • existing_kms_instance_crn (instead of existing_kms_instance_guid)
  • Parse the KMS GUID from existing_kms_instance_crn
  • Parse the KMS region from existing_kms_instance_crn and remove the kms_region variable

Create a simple basic example for the module

Description

All example and the usage section go into a lot of complexity in term of replication, encryption, etc.

It would be desirable to have a simple, basic example that creates an instance with a bucket with the minimum amount of inputs to set.

Review COS DA diagrams

Review the diagram(s) in the reference-architectures directory...

  • Do they contain accurate relevant information
  • Are they using approved icons / objects?

Add support to COS module to create IAM based resource key

The COS module should support creating IAM as well as HMAC resource keys. Currently it only supports HMAC keys however IAM API Keys are the preferred method of authentication for COS as per docs.

Proposal:

Rename these variables (both root level module and fscloud submodule)

  • create_hmac_key -> create_resource_key
  • hmac_key_name -> resource_key_name
  • hmac_key_role -> resource_key_role

Create a new boolean variable called generate-hmac-credentials which should be used like this:

parameters = {
    "serviceid_crn" = var.resource_key_existing_serviceid_crn
    "HMAC"          = var.generate-hmac-credentials
  }

Since IAM is the recommended key type, I think we should set generate-hmac-credentials to false by default (make sure we mention this in the release notes, as it will attempt to destroy any existing hmac credentials and replace it with an IAM credential)

Double check that access tags are indeed evaluated in the CBR rules

Description

There is a need to have an actual check that the tags passed in the CBR rule are indeed used. So far, I think the assumption is that they are, but I don't think this has been actually verified. Perhaps, as part of this add a tag example here: https://github.com/terraform-ibm-modules/terraform-ibm-cos/tree/main/examples

New or affected modules

By submitting this issue, you agree to follow our Code of Conduct

Cannot create COS instance only with skip_iam_authorization_policy=false in v6.0.0

Affected modules

  • COS

Terraform CLI and Terraform provider versions

  • Terraform version: v1.4.5
  • Provider version: v1.51.0

Expected behavior

As a user I should be able to create a COS instance only without buckets and associate it to a KMS instance.

The ibm_iam_authorization_policy to authorize COS instance to read KMS keys should not be skipped.

I need something similar to this:
module "cos_instance" {
source = "../../"
cos_instance_name = "${var.prefix}-cos"
create_cos_bucket = false
resource_group_id = module.resource_group.resource_group_id
existing_kms_instance_guid = module.key_protect_all_inclusive.key_protect_guid
region = var.region
cross_region_location = null
activity_tracker_crn = null
resource_key_existing_serviceid_crn = ibm_iam_service_id.resource_key_existing_serviceid.crn
skip_iam_authorization_policy = false
}

Actual behavior

I get the following error:
Error: Invalid combination of arguments

with module.cos_module["vpc-cos"].ibm_iam_authorization_policy.policy[0],
on .terraform/modules/cos_module/main.tf line 73, in resource "ibm_iam_authorization_policy" "policy":
73: target_service_name = local.kms_service

"target_service_name": one of resource_attributes,target_service_name must be specified

Steps to reproduce (including links and screen captures)

By submitting this issue, you agree to follow our Code of Conduct

Complete example does not execute when region is set to eu-gb

╷
│ Error: InvalidRequest: The region provided in the KMS key CRN does not provide cross region support.
│ status code: 400, request id: a6d17a58-f715-479e-bb0c-1fe53a149f00, host id:
│
│ with module.cos_bucket2.ibm_cos_bucket.cos_bucket[0],
│ on ../../main.tf line 77, in resource "ibm_cos_bucket" "cos_bucket":
│ 77: resource "ibm_cos_bucket" "cos_bucket" {
│

Automated test suites for CBR rules

Description

Extracted out from original issue at #201 .

See whether we could expend the automated test suite as well to catch this type of issue - eg: trying to list the cos bucket in the instance as part of the test executing the complete example would catch this type of problem.

Support creation of resource key with "NONE" role

This request is from BNPP

We want to create a resource_key with role "None" with Terraform.
This is impossible as role seems to not exist:

[17:01] ABDELKRIM Habib
id: terraform-25711290
2024/05/16 14:50:54 Terraform apply | summary: '[ERROR] Error creating resource key when get role:
2024/05/16 14:50:54 Terraform apply | RoleDoesnotExist: None
2024/05/16 14:50:54 Terraform apply | was not found. Valid roles are Writer, Reader, Manager,
2024/05/16 14:50:54 Terraform apply | Content Reader, Object Reader, Object Writer, Service Configuration Reader, Viewer,
2024/05/16 14:50:54 Terraform apply | Administrator, Operator, Editor, Key Manager'

More details refer : IBM-Cloud/terraform-provider-ibm#5391

They want to extend this support for this module also

Don't output the bucket name until bucket is created

Can we have the module only output the bucket name once the bucket is actually created and ready for use? That way consuming modules won't have to add explicit depends_on in their code since they always take bucket name is as input as opposed to bucket ID.

To do this, we can add a depends_on in the outout:

output "bucket_name" {
  description = "Bucket name"
  value       = local.bucket_name
  depends_on  = [local.bucket_id]
}

Authorization COS - KMS malformed

Hello there,

We are facing an issue with your module, especially regarding authorization as shown below:

 ibm_resource_instance.kms_instance: Creating...
 2023/01/27 22:03:52 Terraform apply | module.cos.ibm_resource_instance.cos_instance[0]: Creating...
 2023/01/27 22:04:02 Terraform apply | ibm_resource_instance.kms_instance: Still creating... [10s elapsed]
 2023/01/27 22:04:02 Terraform apply | module.cos.ibm_resource_instance.cos_instance[0]: Still creating... [10s elapsed]
 2023/01/27 22:04:06 Terraform apply | ibm_resource_instance.kms_instance: Creation complete after 14s [id=crn:v1:bluemix:public:kms:eu-fr2:a/3f2cae45b0644f6d87aefcf404f5987f:ba24f111-f80e-4298-a301-ca20371a0d8d::]
 2023/01/27 22:04:06 Terraform apply | data.ibm_kms_keys.encryption_data: Reading...
 2023/01/27 22:04:06 Terraform apply | ibm_kms_key.encryption_key: Creating...
 2023/01/27 22:04:07 Terraform apply | data.ibm_kms_keys.encryption_data: Read complete after 0s [id=ba24f111-f80e-4298-a301-ca20371a0d8d]
 2023/01/27 22:04:07 Terraform apply | ibm_kms_key.encryption_key: Creation complete after 1s [id=crn:v1:bluemix:public:kms:eu-fr2:a/3f2cae45b0644f6d87aefcf404f5987f:ba24f111-f80e-4298-a301-ca20371a0d8d:key:8c87060a-6993-47b5-9ca6-1e2738973927]
 2023/01/27 22:04:08 Terraform apply | module.cos.ibm_resource_instance.cos_instance[0]: Creation complete after 16s [id=crn:v1:bluemix:public:cloud-object-storage:global:a/3f2cae45b0644f6d87aefcf404f5987f:d20bf547-6b85-4f4b-877c-1475c4bde1ee::]
 2023/01/27 22:04:08 Terraform apply | module.cos.ibm_resource_key.resource_key[0]: Creating...
 2023/01/27 22:04:08 Terraform apply | module.cos.ibm_iam_authorization_policy.policy[0]: Creating...
 2023/01/27 22:04:09 Terraform apply | module.cos.ibm_iam_authorization_policy.policy[0]: Creation complete after 1s [id=a77acd3f-d3c4-4c63-b1b5-0951bfee108d]
 2023/01/27 22:04:09 Terraform apply | module.cos.ibm_cos_bucket.cos_bucket[0]: Creating...
 2023/01/27 22:04:12 Terraform apply | module.cos.ibm_resource_key.resource_key[0]: Creation complete after 4s [id=crn:v1:bluemix:public:cloud-object-storage:global:a/3f2cae45b0644f6d87aefcf404f5987f:d20bf547-6b85-4f4b-877c-1475c4bde1ee:resource-key:3c34a882-8178-4ab8-b136-c010847e62b5]
 2023/01/27 22:04:12 Terraform apply | 
 2023/01/27 22:04:12 Terraform apply | Error: ServiceNotAuthorized: The specified COS Service Instance does not have sufficient permissions to access the resource associated with the KMS key CRN.
 2023/01/27 22:04:12 Terraform apply | 	status code: 401, request id: d32b07a0-e350-4e6a-aa86-787e467c0b4f, host id: 
 2023/01/27 22:04:12 Terraform apply | 
 2023/01/27 22:04:12 Terraform apply |   with module.cos.ibm_cos_bucket.cos_bucket[0],
 2023/01/27 22:04:12 Terraform apply |   on .terraform/modules/cos/main.tf line 76, in resource "ibm_cos_bucket" "cos_bucket":
 2023/01/27 22:04:12 Terraform apply |   76: resource "ibm_cos_bucket" "cos_bucket" {
 2023/01/27 22:04:12 Terraform apply | 
 2023/01/27 22:04:12 �[1m�[31mTerraform APPLY error: Terraform APPLY errorexit status 1�[39m�[0m

We found out that your authorization resource input for the source_service_id, is not correct, it should be the GUID of the COS instance instead of ID.

It results into this authorization in your IAM:

image

###Proposal

locals {
 cos_instance_id      = var.create_cos_instance == true ? tolist(ibm_resource_instance.cos_instance[*].id)[0] : var.existing_cos_instance_id
 cos_instance_guid      = var.create_cos_instance == true ? : tolist(ibm_resource_instance.cos_instance[*].guid)[0] : element(split(":",var.existing_cos_instance_id),length(split(":",var.existing_cos_instance_id)-3)
 create_access_policy = var.encryption_enabled && var.create_cos_instance
}

# Create IAM Access Policy to allow Key protect to access COS instance
resource "ibm_iam_authorization_policy" "policy" {
 count                       = local.create_access_policy ? 1 : 0
 source_service_name         = "cloud-object-storage"
 source_resource_instance_id = local.cos_instance_guid
 target_service_name         = "kms"
 target_resource_instance_id = var.existing_key_protect_instance_guid
 roles                       = ["Reader"]
}

Resulting into a correct Authorization:
image

Let me know if you want me to do a PR.

Regards

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.