terraform-ibm-modules/terraform-ibm-watsonx-saas-da

A deployable architecture solution to deploy IBM Watsonx SaaS resources.

Watsonx.ai SaaS with Assistant and Governance Deployable Architecture

The Watsonx.ai SaaS with Assistant and Governance Deployable Architecture is designed to automate the deployment and configuration of the IBM watsonx platform in an IBM Cloud account. The IBM watsonx platform is made of several services working together to offer AI capabilities to end users, who can explore them using IBM watsonx projects.

In addition, this deployable architecture configures a starter project for an IBM Cloud user.

Overview

• terraform-ibm-watsonx-saas-da
• Examples
  • Basic example
  • Complete example
• Contributing

terraform-ibm-watsonx-saas-da

The solution supports the following:

• Creating a new resource group, or using an existing one.
• Provisioning the following services:
  • Watson Machine Learning
  • Watson Studio
  • Cloud Object Storage.
• Configuring the IBM watsonx profile and creating a starter IBM watsonx project. for an IBM Cloud user, who becomes the admin of the IBM watsonx project.

As result the IBM watsonx admin can log into IBM watsonx in the target account and start experimenting with the starter project.

Optionally, the solution supports:

• Enabling the storage delegation for the provisioned Cloud Object Storage instance using your own encryption keys with Key Protect.
• Provisioning of one or more of the services, with a selectable service plan:
  • watsonx.data
  • watsonx.governance
  • watsonx Assistant
  • Watson Discovery.

Required IAM access policies

The following permissions are required to deploy this solution.

• Administrator role on All Account Management services to create a new resource group, and to enable storage delegation for the Cloud Object Storage instance.
• Manager service role on the Key Protect instance used for storage delegation.
• Editor platform role on Watson Machine Learning to create and delete the service.
• Editor platform role on Watson Studio to create or delete the service.
• Editor platform role on Cloud Object Storage to create and delete the service.
• Editor platform role on watsonx.data if you must provision.
• Editor platform role on watsonx.governance if you must provision.
• Editor platform role on watsonx Assistant if you must provision.
• Editor platform role on Watson Discovery if you must provision.

The IBM watsonx administrator needs the following permissions:

• Administrator role on All Account Management services.
• Administrator role on All Identity and Access enabled services.
• Manager service role on Cloud Object Storage to create service credentials. That is not needed if you configure storage delegation.

You can use the IBM provided IAM Access Group Terraform Module to configure deployers and watsonx admins access groups and add members to them.

Requirements

Name	Version
terraform	>= 1.5.0
ibm	>= 1.66.0
restapi	>= 1.19.1

Modules

Name	Source	Version
configure_project	./configure_project	n/a
configure_user	./configure_user	n/a
cos	terraform-ibm-modules/cos/ibm//modules/fscloud	8.10.7
resource_group	terraform-ibm-modules/resource-group/ibm	1.1.6
storage_delegation	./storage_delegation	n/a

Resources

Name	Type
ibm_resource_instance.assistant_instance	resource
ibm_resource_instance.data_instance	resource
ibm_resource_instance.discovery_instance	resource
ibm_resource_instance.governance_instance	resource
ibm_resource_instance.machine_learning_instance	resource
ibm_resource_instance.studio_instance	resource
ibm_iam_auth_token.restapi	data source
ibm_resource_instance.existing_assistant_instance	data source
ibm_resource_instance.existing_data_instance	data source
ibm_resource_instance.existing_discovery_instance	data source
ibm_resource_instance.existing_governance_instance	data source
ibm_resource_instance.existing_machine_learning_instance	data source
ibm_resource_instance.existing_studio_instance	data source

Inputs

Name	Description	Type	Default	Required
cos_kms_crn	Key Protect service instance CRN used to encrypt the COS buckets used by the watsonx projects.	string	null	no
cos_kms_key_crn	Key Protect key CRN used to encrypt the COS buckets used by the watsonx projects. If not set, then the cos_kms_new_key_name must be specified.	string	null	no
cos_kms_new_key_name	Name of the Key Protect key to create for encrypting the COS buckets used by the watsonx projects.	string	""	no
cos_kms_ring_id	The identifier of the Key Protect ring to create the cos_kms_new_key_name into. If it is not set, then the new key will be created in the default ring.	string	null	no
cos_plan	The plan that's used to provision the Cloud Object Storage instance.	string	"standard"	no
existing_assistant_instance	CRN of the an existing watsonx Assistance instance.	string	null	no
existing_data_instance	CRN of the an existing watsonx.data instance.	string	null	no
existing_discovery_instance	CRN of the an existing Watson Discovery instance.	string	null	no
existing_governance_instance	CRN of the an existing watsonx.governance instance.	string	null	no
existing_machine_learning_instance	CRN of the an existing Watson Machine Learning instance.	string	null	no
existing_studio_instance	CRN of the an existing Watson Studio instance.	string	null	no
ibmcloud_api_key	The API key that's used with the IBM Cloud Terraform IBM provider.	string	n/a	yes
location	The location that's used with the IBM Cloud Terraform IBM provider. It's also used during resource creation.	string	"us-south"	no
resource_group_name	The name of a new or an existing resource group where the resources are created.	string	n/a	yes
resource_prefix	The name to be used on all Watson resources as a prefix.	string	"watsonx-poc"	no
use_existing_resource_group	Determines whether to use an existing resource group.	bool	false	no
watson_discovery_plan	The plan that's used to provision the Watson Discovery instance.	string	"do not install"	no
watson_machine_learning_plan	The plan that's used to provision the Watson Machine Learning instance.	string	"v2-standard"	no
watson_studio_plan	The plan that's used to provision the Watson Studio instance. The plan you choose for Watson Studio affects the features and capabilities that you can use.	string	"professional-v1"	no
watsonx_admin_api_key	The API key of the IBM watsonx administrator in the target account. The API key is used to configure the user and the project.	string	null	no
watsonx_assistant_plan	The plan that's used to provision the watsonx Assistance instance.	string	"do not install"	no
watsonx_data_plan	The plan that's used to provision the watsonx.data instance.	string	"do not install"	no
watsonx_governance_plan	The plan used to provision the watsonx.governance instance. The available plans depend on the region where you are provisioning the service from the IBM Cloud catalog.	string	"do not install"	no
watsonx_project_description	A description of the watson project that's created by the WatsonX.ai SaaS Deployable Architecture.	string	"Watson project created by the watsonx-ai SaaS deployable architecture."	no
watsonx_project_name	The name of the watson project.	string	"demo"	no
watsonx_project_tags	A list of tags associated with the watsonx project. Each tag consists of a single string containing up to 255 characters. These tags can include spaces, letters, numbers, underscores, dashes, as well as the symbols # and @.	list(string)	[ "watsonx-ai-SaaS" ]	no

Outputs

Name	Description
resource_group_id	The resource group ID that's used to provision the resources.
watson_discovery_crn	The CRN of the Watson Discovery instance.
watson_discovery_dashboard_url	The dashboard URL of the Watson Discovery instance.
watson_discovery_guid	The GUID of the Watson Discovery instance.
watson_discovery_name	The name of the Watson Discovery instance.
watson_discovery_plan_id	The plan ID of the Watson Discovery instance.
watson_machine_learning_crn	The CRN of the Watson Machine Learning instance.
watson_machine_learning_dashboard_url	The dashboard URL of the Watson Machine Learning instance.
watson_machine_learning_guid	The GUID of the Watson Machine Learning instance.
watson_machine_learning_name	The name of the Watson Machine Learning instance.
watson_machine_learning_plan_id	The plan ID of the Watson Machine Learning instance.
watson_studio_crn	The CRN of the Watson Studio instance.
watson_studio_dashboard_url	The dashboard URL of the Watson Studio instance.
watson_studio_guid	The GUID of the Watson Studio instance.
watson_studio_name	The name of the Watson Studio instance.
watson_studio_plan_id	The plan ID of the Watson Studio instance.
watsonx_assistant_crn	The CRN of the watsonx Assistant instance.
watsonx_assistant_dashboard_url	The dashboard URL of the watsonx Assistant instance.
watsonx_assistant_guid	The GUID of the watsonx Assistant instance.
watsonx_assistant_name	The name of the watsonx Assistant instance.
watsonx_assistant_plan_id	The plan ID of the watsonx Assistant instance.
watsonx_data_crn	The CRN of the watsonx.data instance.
watsonx_data_dashboard_url	The dashboard URL of the watsonx.data instance.
watsonx_data_guid	The GUID of the watsonx.data instance.
watsonx_data_name	The name of the watsonx.data instance.
watsonx_data_plan_id	The plan ID of the watsonx.data instance.
watsonx_governance_crn	The CRN of the watsonx.governance instance.
watsonx_governance_dashboard_url	The dashboard URL of the watsonx.governance instance.
watsonx_governance_guid	The GUID of the watsonx.governance instance.
watsonx_governance_name	The name of the watsonx.governance instance.
watsonx_governance_plan_id	The plan ID of the watsonx.governance instance.
watsonx_platform_endpoint	The endpoint of the watsonx platform.
watsonx_project_bucket_name	The name of the COS bucket created by the watsonx project.
watsonx_project_id	The ID watsonx project that's created.
watsonx_project_location	The location watsonx project that's created.
watsonx_project_url	The URL of the watsonx project that's created.

Contributing

You can report issues and request features for this module in GitHub issues in the module repo. See Report an issue or request a feature.

To set up your local development environment, see Local development setup in the project documentation.

terraform-ibm-watsonx-saas-da's Issues

Include resource group ID as output

Description

Include created resource group ID as output - as discussed, we will probably need this in the RAG App DA later.

New or affected modules

terraform-ibm-watsonx-saas-da

---

Storage delegation does not work when KMS is private

When KeyProtect is set to private endpoints/network only, the storage-delegation module will fail to create a key.

This can be resolved using the endpoint_type parameter for ibm_kms_key resources.

As this DA will typically run in Schematics, and KeyProtect private endpoints are always enabled, I think you can just add the parameter, but if you want to add logic/variables to control it, that is up to you.

Set paid plans as default

Free plans have some restrictions, we should use paid plans to enable all the capabilities.

Align references to watson and watsonx products in the output variables to their proper brand names.

Affected modules

• watsonx_saas

Terraform CLI and Terraform provider versions

• Terraform version:
• Provider version:

Terraform output

Debug output

Expected behavior

all Brand names should be accurate.
Watsonx Discovery -> Watson Discovery
watson platform -> watsonx platform
Watson Governance -> watsonx.governance

Actual behavior

Steps to reproduce (including links and screen captures)

1. Run terraform apply

Anything else

---

By submitting this issue, you agree to follow our Code of Conduct

Update status badge to graduated on README.md

Allow provisioning of Watson instances with private endpoints

Description

The user should be able to enforce the allowed networks on provisioned instances by allowing to pass the service-endpoints parameter when creating a service instance:

 resource "ibm_resource_instance" "machine_learning_instance" {
......
   parameters = {
     service-endpoints: "private"
   }

The default for the service endpoint parameter could be set to "public" to keep the existing default functionality.
Note that the "private" option is not available on "lite" plans.

Since switching to private only endpoints may disrupt the integration or other processes, we need to have a way to control endpoint type for each service individually.

Private endpoints for Watson Machine Learning instances are required by the AI Guardrails compliance profile in SCC.
The endpoint type cannot be changed once the instance is provisioned.

New or affected modules

---

By submitting this issue, you agree to follow our Code of Conduct

onboard watsonx da into catalog from release in this repo

Description

@ocofaigh - could we plug the catalog onboarding pipeline here?

use regional data platform APIs

Trying to deploy the SaaS DA in eu-de region, I found a strange behaviour. Basically I provisioned ML and Studio in eu-de (COS is global) but when I called POST /transactional/v2/projects , the watson project was created in us-south (and the COS bucket as well). If I try to use it, it does not work since there is no ML instance associated.

The issue is in the endpoint used, http://api.dataplatform.cloud.ibm.com/ is not global but is related to us-south.
For common data platform APIs:

• Dallas - api.dataplatform.cloud.ibm.com
• London - api.eu-uk.dataplatform.cloud.ibm.vom
• Frankfurt - api.eu-de.dataplatform.cloud.ibm.com
• Tokyo - api.jp-tok.dataplatform.cloud.ibm.com

SaaS DA is intermittently failing on deploy

The RAG stack failed catalog validation on the Watson Saas deploy. After rerun the validation passed.
It is an intermittent failure

error

Terraform planned the following actions, but then encountered a problem:

  # module.configure_user.null_resource.configure_user must be replaced
-/+ resource "null_resource" "configure_user" {
      ~ id       = "2144875551820406457" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "always_run" = "2024-07-04T17:57:35Z" -> (known after apply)
        }
    }

  # module.configure_user.null_resource.restrict_access must be replaced
-/+ resource "null_resource" "restrict_access" {
      ~ id       = "7300154366233028698" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "always_run" = "2024-07-04T17:57:34Z" -> (known after apply)
        }
    }

Plan: 2 to add, 0 to change, 2 to destroy.

Error: failed to find an object with the 'metadata/guid' key = 'f2d50750-891d-4c71-937e-6bad51cb55b0' at //api.dataplatform.cloud.ibm.com/v2/projects

  with module.configure_project[0].data.restapi_object.get_project,
  on configure_project/main.tf line 56, in data "restapi_object" "get_project":
  56: data "restapi_object" "get_project" {

 [1m�[31mTerraform PLAN error: Terraform PLAN errorexit status 1�[39m�[0m

resource group handling

terraform-ibm-watsonx-saas-da/main.tf

Line 14 in ea3f756

module "resource_group" {

Replace by

module "resource_group" {
  source                       = "terraform-ibm-modules/resource-group/ibm"
  version                      = "1.1.5"
  resource_group_name          = var.existing_resource_group == false ? var.resource_group_name : null
  existing_resource_group_name = var.existing_resource_group == true ? var.resource_group_name : null
}

for consistency.

The boolean also allows to execute the logic in the case where group name is not known at apply time (eg: when terraform logic call the DA)

Error when requesting no project to be created

When running the DA with watsonx_project_name = "__NULL__" the terraform plan fails due to no items in module.configure_project.

The DA should allow passing empty project name which should result in no WML project being created.
The outputs related to WML project should be empty in this case.

terraform-ibm-watsonx-saas-da/outputs.tf

Line 132 in 1b805c5

value = module.configure_project[0].watsonx_project_id

split api key used for provisioning and configuration

currently we use ibmcloud_api_key for everything, we should split what is used from TF provider to provision resources and what we use to call watson APIs. In this way we can use a trusted profile to provision the services.

Allow to use references to existing instances

Description

When the Watson SaaS DA is used to deploy resources for multiple solutions or is used in a stack context, the users should be able to use references to existing instances instead of provisioning new ones, especially for services with expensive plans like Discovery or Assistant.

• Allow an optional parameter for Watson Discovery instance GUID. If supplied, no new instance is created, the plan parameter is ignored. The output values related to Discovery instance are set to corresponding attributes of the instance
• Allow an optional parameter for Watson Assistant instance GUID. If supplied, no new instance is created, the plan parameter is ignored. The output values related to Discovery instance are set to corresponding attributes of the instance

The main advantage of this feature is the ability to seamlessly integrate the existing instances in stack deployments where the dependent components can use the output of the Watson SaaS DA in the same way whether new instances are provisioned or existing ones are referenced.

New or affected modules

mail.tf

---

By submitting this issue, you agree to follow our Code of Conduct

Decoding access token fails intermittently

terraform-ibm-watsonx-saas-da/main.tf

Line 56 in db51d5c

account_id = jsondecode(base64decode(regex("^Bearer .+\\.(.+)\\..+$", data.ibm_iam_auth_token.deployer.iam_access_token)[0])).account.bss

Terraform's base64decode function is sensitive to base64 padding and the JWT token does not have one.
When the token part has certain length that would require the padding (trailing ==) in the encoded value, base64decode throws an error and plan/apply fails.
Apparently parsing the token for an account ID is not a good idea.

I think it would be better to fetch the necessary attributes in the configure_project module that should construct the URL and return it. Account ID can be parsed from a CRN of a watson instance.

demo project already exists - initial run

Seen this error a few times when deploying the DA in an existing account. It seems that the project name is unique per account (regardless of the number of machine learning instance), so suggesting to add a random suffix to the name to avoid clashes.

configure_project | Failed | unexpected response code '400': Project name 'demo' already used.

Add support for HPCS key encryption

The cos_kms_crn input has validation that will only allow Key Protect, however HPCS is supported by the COS service for encrypting buckets.
By not supporting HPCS, it means the GenAI / RAG stack cannot support HPCS, yet all other DAs allow HPCS.

I also notice the code is using the provider directly to create keys and key rings. All other DAs are using https://github.com/terraform-ibm-modules/terraform-ibm-kms-all-inclusive

Support deploying some of the watson services in jp-tok region.

Description

Use case: some customers want to deploy some of the watson services in jp-tok region.

Issue: currently, the location field is set to allow only the regions that are supported by ALL watson service.

2 approaches:

• In the location field - add all regions supported by at least one watson service. Add some static checks in the terraform logic to error out if the users selected to deploy a service in a region that is not supported
• Or, add an optional region input for each watson service

Split user and project configuration into separate modules

Description

As we need to create a new user (to be confirmed) and a new project (this is confirmed) in the RAG App DA, it makes sense to reuse the same modules used to provision WatsonX AI users/projects in the terraform-ibm-watsonx-saas-da module. Therefore, could we split those modules into their own repositories so they can be reused?

New or affected modules

terraform-ibm-watsonx-saas-da

new:
terraform-ibm-watsonx-saas-user
terraform-ibm-watsonx-saas-project

---

By submitting this issue, you agree to follow our Code of Conduct

Improve description of the variables

#4 (comment)