hashicorp/terraform-helm

Helm chart to install Terraform Cloud Operator and other associated components.

Terraform Cloud Operator Helm Chart

This repository contains the official HashiCorp Helm chart for installing and configuring the Terraform Cloud Operator on Kubernetes. This chart supports multiple use cases of Terraform on Kubernetes depending on the values provided.

This chart is hosted on the official HashiCorp helm chart repository.

Prerequisites

To use the charts here, Helm must be installed in your Kubernetes cluster. Setting up Kubernetes and Helm and is outside the scope of this README. Please refer to the Kubernetes and Helm documentation.

The versions required are:

• Helm +3.0.1 - This is the earliest version of Helm tested. It is possible it works with earlier versions but this chart is untested for those versions.
• Kubernetes 1.15+ - This is the earliest version of Kubernetes tested. It is possible that this chart works with earlier versions but it is untested.

In addition to Helm, you must also have a:

• Terraform Cloud organization - Create an organization on Terraform Cloud/Enterprise.
• Terraform Cloud Team API Token - Generate a team API token for the Terraform Cloud organization you want to use. Make sure the team at least has privileges to manage workspaces.

Usage

Before installing the chart, you must create two Kubernetes secrets:

1. credentials file contents with Terraform Cloud Team API token. See Terraform Cloud Configuration File Syntax for proper format.

   $ kubectl -n $NAMESPACE create secret generic terraformrc --from-file=credentials

2. Sensitive variables for a workspace.

   $ kubectl -n $NAMESPACE create secret generic workspacesecrets --from-literal=secret_key=abc123

To use the charts, you must add the HashiCorp Helm Chart repository.

$ helm repo add hashicorp https://helm.releases.hashicorp.com
$ helm search repo hashicorp/terraform
$ helm install --namespace ${RELEASE_NAMESPACE} hashicorp/terraform --generate-name

NAME               	CHART VERSION	APP VERSION	DESCRIPTION
hashicorp/terraform	1.0  	           	Install and configure Terraform Cloud Operator ...

NAME: terraform-1589480669
LAST DEPLOYED: Thu May 14 11:24:32 2020
NAMESPACE: operator
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Terraform Cloud Operator!

Now that you have deployed HashiCorp Terraform Cloud Operator, you should look over the docs on using
Terraform with Kubernetes available here:

https://github.com/hashicorp/terraform-k8s/blob/master/README.md


Your release is named terraform-1589480669. To learn more about the release, try:

  $ helm status terraform-1589480669
  $ helm get terraform-1589480669

Please see the many options supported in the values.yaml file.

To create a Terraform workspace, you can create a separate Helm chart to deploy the custom resource or examine the example under example/. Helm does not currently support a wait function before deletion, which will cause custom resources to remain behind.

Note that the Helm chart automatically installs all Custom Resource Definitions under the crds/ directory. As a result, any updates to the schema must be manually copied into the directory and removed from the Kubernetes cluster:

$ kubectl delete crd workspaces.app.terraform.io

If the CRD is not updated correctly, you will not be able to create a Workspace Custom Resource.

terraform-helm's Issues

RBAC should be more configurable

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Pulling an issue from here -> : hashicorp/terraform-k8s#99

RBAC is too permissive and should be configurable to the use case. I understand the need for some of these permissions if I were controlling k8s applications inside this k8s cluster, however I have another use case where its not managing k8s apps, and thus I'd like to remove those permissions.

- apiGroups:
  - apps
  resources:
  - deployments
  - daemonsets
  - replicasets
  - statefulsets
  verbs:
  - '*'

Also, there is no documentation on why it needs certain roles. Maybe thats another ticket?

Potential Helm Configuration

{{- if $managingKubernetesApps }}
- apiGroups:
  - apps
  resources:
  - deployments
  - daemonsets
  - replicasets
  - statefulsets
  verbs:
  - '*'
 {{- end }}

References

https://github.com/hashicorp/terraform-helm/blob/master/templates/sync-workspace-role.yaml

Terraform Cloud Operator for Kubernetes Drift detection and remediation

1. Curious to know if this operator can detect drift in remote state on TFE workspace as well on the original aws account when sqs from example is deleted by someone and perform auto remediation as per k8s controller mechanism.

2. How can we pass output of one TF module resource as input to other when created as yaml files?

Support imagePullSecrets for offline private container registries

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

There is no direct helm way to specify imagePullSecrets

Potential Helm Configuration

global:
  imagePullSecrets:
  - nexus-secret # .i.e.

where nexus-secret (for example) is already in existing secret with dockerconfig type.

References

https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod

Rename 'master' branch to 'main'

Description

Description

Rename the master branch to main per HashiCorp's inclusive language commitment.

See this discussion for more information.

Installing Operator to Cluster fails with runAsNonRoot issue

Helm install of this operator via terraform into AWS EKS cluster gives the following error. Same error is encountered when using helm directly:
Warning Failed 41m (x12 over 43m) kubelet Error: container has runAsNonRoot and image has non-numeric user (nobody), cannot verify user is non-root (pod: "terraform-1640001071-terraform-sync-workspace-579fb5c5bc-npf59_tfcloud(0997638a-cb58-43dd-9d0b-13722df1bce9)", container: terraform-sync-workspace)

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

terraform-k8s, Helm, & Kubernetes Version

Version 1.1.0, Terraform 1.0.7, Helm provider 2.4.0, helm version 3.7.2, Kubernetes 1.21

Helm Values

Defaults used

• #0000

`terraform-helm` chart should appear on helm.releases.hashicorp.com

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

terraform-k8s, Helm, & Kubernetes Version

Affected Resource(s)

helm.releases.hashicorp.com

Helm Values

# Copy-paste your Helm values.yaml here.

Debug Output

Expected Behavior

The terraform-helm chart should appear

Actual Behavior

Steps to Reproduce

Important Factoids

The chart appears correctly via command line, without requiring --devel

References

• #0000

seperating image.tag from image.repository is a good practice

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

terraform-helm/values.yaml

Line 14 in e90fb3a

imageK8S: "hashicorp/terraform-k8s:1.0.0"

this line shows that the image value is not divided into repository and tag.
Dividing image value into 2 parts has the following advantages:

1. Ability to override repository value (if i have proxy registry) while relying on the default value of "tag"
2. within the same chart, Defaulting "tag" (if not assigned a value from user) to .Chart.AppVersion.
3. being consistent with community charts

Potential Helm Configuration

global:
  imageK8S:
     repository: hashicorp/terraform-k8s
     tag: 1.0.0 # better make it empty string, and put this value under Chart.yaml ( appVersion)

References

terraform-helm/values.yaml

Line 14 in e90fb3a

imageK8S: "hashicorp/terraform-k8s:1.0.0"

Support installing Private Terraform Enterprise ( self-hosted) on top of kubernetes

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

As far as i understand, this helm chart assumes that you have already Terraform Cloud or Enterprise, and then, it starts from that.
We would like to use this helm chart to also provision a totally offline and private Terraform enterprise.
Otherwise, is there another chart does that?

Potential Helm Configuration

privateTerraformEnterprise:
   install: false
   image:
     repository: 
     tag: ...

References

https://youtu.be/ynXFwKYwu4o?t=1380

Use of k8WatchNamespace (helm value parameter) leads to RBAC error

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

terraform-k8s, Helm, & Kubernetes Version

terraform-k8s: hashicorp/terraform-k8s:1.0.0
Helm: v3.6.0
Kubernetes: v1.19.8-eks-96780e

Helm Values

syncWorkspace:
  k8WatchNamespace: dev

Debug Output

E0616 16:19:10.725113 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1alpha1.Workspace: workspaces.app.terraform.io is forbidden: User "system:serviceaccount:terraform-operator:terraform-operator-terraform-sync-workspace" cannot list resource "workspaces" in API group "app.terraform.io" in the namespace "dev"

Expected Behavior

Terraform Cloud Operator is capable of watch and write secrets to the k8WatchNamespace defined namespace

Actual Behavior

Terraform Operator logs that have no permissions for access resource "workspaces" in API group "app.terraform.io" in the namespace "dev".

Steps to Reproduce

1. Use provided Helm values and install the operator in the "operator-terraform" namespace.

2. Apply the following YAML via kubectl apply (please note that the namespace is dev):

apiVersion: app.terraform.io/v1alpha1
kind: Workspace
metadata:
  name: infra-dev
  namespace: dev
spec:
  organization: Bla
  secretsMountPath: "/tmp/secrets"
  module:
    source: "terraform-aws-modules/sqs/aws"
    version: "3.1.0"
  outputs:
    ......
  variables:
    ......

CRD uses apiextensions.k8s.io/v1beta1, which is deprecated in 1.22+

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

terraform-k8s, Helm, & Kubernetes Version

terraform-k8s - 1.1.0
helm - 3.8.0
kubernetes - 1.22.3-gke.1500

Affected Resource(s)

(https://github.com/hashicorp/terraform-helm/blob/master/crds/app.terraform.io_workspaces_crd.yaml)

Helm Values

Default helm values.

Debug Output

Last Helm logs: Normal error 12s helm-controller reconciliation failed: Helm install failed: failed to install CRD crds/app.terraform.io_workspaces_crd.yaml: unable to recognize "": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"

Expected Behavior

Helm chart installs the CRD

Actual Behavior

Helm chart errors on CRD install

Steps to Reproduce

Install the helm chart on k8s 1.22+

Important Factoids

References

https://kubernetes.io/docs/reference/using-api/deprecation-guide/#customresourcedefinition-v122

Workspace Custom Resource Chart

This is to add a Helm chart to deploy the Workspace Custom Resource. Currently, we have examples listed under the example directory as Kubernetes YAML.

terraform-k8s image v1.1.1 requires additional role rules

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

terraform-k8s, Helm, & Kubernetes Version

terraform-k8s: 1.1.1
Helm: 3.8.0
Kubernetes: 1.21.4

Affected Resource(s)

https://github.com/hashicorp/terraform-helm/blob/master/templates/sync-workspace-role.yaml

Helm Values

global:
  imageK8S: "hashicorp/terraform-k8s:1.1.1"

Debug Output

E0301 12:31:04.773294       1 leaderelection.go:325] error retrieving resource lock k8s-tfe-test/4ebc9a03.terraform.io: leases.coordination.k8s.io "4ebc9a03.terraform.io" is forbidden: User "system:serviceaccount:k8s-tfe-test:operator-test-terraform-sync-workspace" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "k8s-tfe-test"

Expected Behavior

Helm chart should define all the required roles/rolebindings for the serviceaccount used by the image.

Actual Behavior

get and update permissions are missing for coordination.k8s.io : leases for the serviceaccount created as part of the helm chart.

Steps to Reproduce

Helm install with the new image.

Important Factoids

I know that this git repo is still using the older version (v1.1.0). Hopefully it can be updated to use new version and include additional roles.

References

Add port to pod to allow prometheus-operator PodMonitor

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Add optional port definition to deployment's pod spec to allow prometheus-operator PodMonitor to scrape the /metrics endpoint.

Potential Helm Configuration

          {{- if .Values.syncWorkspace.exposeMetricsPort }}
          ports:
            - containerPort: 8383
              protocol: TCP
          {{- end }}

References