ericchiang/terraform-provider-k8s

Kubernetes Terraform provider with support for raw manifests

This project has been archived and is unmaintained.

Kubernetes Terraform Provider

The k8s Terraform provider enables Terraform to deploy Kubernetes resources. Unlike the official Kubernetes provider it handles raw manifests, leveraging kubectl directly to allow developers to work with any Kubernetes resource natively.

Usage

Use go get to install the provider:

go get -u github.com/ericchiang/terraform-provider-k8s

Register the plugin in ~/.terraformrc:

providers {
  k8s = "/$GOPATH/bin/terraform-provider-k8s"
}

The provider takes the following optional configuration parameters:

• If you have a kubeconfig available on the file system you can configure the provider as:

provider "k8s" {
  kubeconfig = "/path/to/kubeconfig"
}

• If you content of the kubeconfig is available in a variable, you can configure the provider as:

provider "k8s" {
  kubeconfig_content = "${var.kubeconfig}"
}

WARNING: Configuration from the variable will be recorded into a temporary file and the file will be removed as soon as call is completed. This may impact performance if the code runs on a shared system because and the global tempdir is used.

The k8s Terraform provider introduces a single Terraform resource, a k8s_manifest. The resource contains a content field, which contains a raw manifest.

variable "replicas" {
  type    = "string"
  default = 3
}

data "template_file" "nginx-deployment" {
  template = "${file("manifests/nginx-deployment.yaml")}"

  vars {
    replicas = "${var.replicas}"
  }
}

resource "k8s_manifest" "nginx-deployment" {
  content = "${data.template_file.nginx-deployment.rendered}"
}

In this case manifests/nginx-deployment.yaml is a templated deployment manifest.

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: ${replicas}
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

The Kubernetes resources can then be managed through Terraform.

$ terraform apply
# ...
Apply complete! Resources: 1 added, 1 changed, 0 destroyed.
$ kubectl get deployments
NAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3         3         3            3           1m
$ terraform apply -var 'replicas=5'
# ...
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
$ kubectl get deployments
NAME               DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   5         5         5            3           3m
$ terraform destroy -force
# ...
Destroy complete! Resources: 2 destroyed.
$ kubectl get deployments
No resources found.

terraform-provider-k8s's Issues

Name, namespace or kind changes not supported

Changing the name, namespace or kind in the content causes the provider to get confused about the resource that it is tracking. The read operations will continue to use the old names, etc... as these are encoded in the id associated with the resource. The creation operations, however, will use a combination of the newer values encoded in the content and the older values encoded in the id.

Ultimately I think the name, namespace and kind should all be part of the terraform resource so that they can be ForceNew. This avoids all of these strange edge cases.

I have a proof of concept implementation here and can get it into an MR if there is interest. It will, however, potentially conflict with #21

Please support terraform imports

This is in follow up to PR #9.

I work at SlingTV. I am on an 8 person DevOps team. We support our developers, which at my estimation amount to something like 250+ engineers. We also support our ops team, who are responsible for the deployment of our backend software.

Our app deployment is large, and relies heavily on VMware, SaltStack, SSH, and even some manual deployment steps currently.

We are also in the early stages of putting our deployments into kubernetes. We already have several apps in k8s.

I would like to bring a large portion of our deployments under management by Terraform. However, if I can't import what is already deployed using old or undesirable technologies into Terraform state without affecting already-live deployments, putting stuff into TF is going to be a hard sell to my ops team. Likewise, if I can't re-create the TF state (if it were ever lost) without the need to touch current deployments, my Ops team probably won't let me introduce a new, never-before-used tool into their environment.

Fortunately, terraform import allows for both use cases as described above. It exactly fits what I need to make sure that I can convince them to use terraform in our broader deployment strategy.

The only plugin that is even close to handling kubernetes deloyments right is, in my opinion, this one. It allows me to use CRDs and I don't have to wait for the the official kubernetes provider to support the workloads API.

I would like terraform import to be supported with this plugin. This way I can make sure I can recover if terraform state were ever deleted without actually touching any live deployments. This will also allow me to write new terraform states, then import what I already have deployed into terraform, again without touching the live deploy.

I would use terraform import like this:

terraform import k8s_manifest.nginx-deployment /apis/extensions/v1beta1/namespaces/default/deployments/releaseme-example

That is, I would give terraform an ID of a live deployment and point to what terraform resource stanza I had written to tell TF what code should manage the live deploy in the future.

All that is required for that to work is to have the provider support what is known as an "id-only refresh", meaning that it has to be able to repopulate all of the attributes of the resource in its resourceManifestRead function. So, if we can change that function to repopulate "content" based on what is actually already deployed, importing will work simply by adding some boilerplate just-use-the-read-function passthrough code when creating the Resource struct and passing to terraform in the provider.

This is the approach I took in #9 . I simply read in from a k8s annotation what was previously given to kubectl in a kubectl apply call. I also normalized the contents of the "contents" value so that diffs between what terraform reads in production and what terraform was given would make sense, and wouldn't just be over something like whitespace or quotes.

design question: shelling out versus k8s REST API

First off, thanks for your work on this project, it's far more useful than the default k8s terraform provider.

Anyway, I'm curious why you chose to shell out to kubectl versus using the kubernetes REST API. You describe why a bit in the README, but I'm curious why you can't use raw manifests w the REST API in a similar fashion? What's the advantage of working with "any Kubernetes resource natively"?

Using Kubernetes API instead of kubectl

Are there any thoughts/plans on using the API [1] directly rather than via kubectl (or rely on locally persisted config/credentials)? This article [2] is interesting in this context.

[1] https://kubernetes.io/docs/concepts/overview/kubernetes-api/
[2] http://k8s.uk/kubectl-vs-http-api.html

provider should allow a custom kubeconfig path

Leading slash in terraformrc provider in README

In the readme, it says to add this block to a ~/.terraformrc

providers {
  k8s = "/$GOPATH/bin/terraform-provider-k8s"
}

I'm curious about that leading slash, is it intentional or a mistake? I did my best to read through the Terraform configuration syntax docs to find out whether it's special syntax for interpolating environment variables, but couldn't find anything.

Allow to select kubectl context in the provider configuration

It seems like right now there’s no specific configuration for the provider itself and it only defines a resource. It should allow to setup a kubectl context somewhere, because assuming that current context is the right one is pretty dangerous.

If it actually allows for that and I missed it please let me know.

provider should allow a custom kubectl path

Make a release in GitHub

Hi, for people like me we are not Go developper and have no access to a Go development environment, can you make a release that we can download in GitHub?

Thanks

apiVersion leaks into resource name

I have a CRD:

# From https://github.com/jetstack/cert-manager/blob/912c7672bd3e6ba1e7ef9dff48083ee6f3d6cbab/contrib/manifests/cert-manager/with-rbac.yaml                                                                                                                                                                                                                                  
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: clusterissuers.certmanager.k8s.io
  annotations:
    "helm.sh/hook": crd-install
  labels:
    app: cert-manager
    chart: cert-manager-v0.6.0-dev.1
    release: cert-manager
    heritage: Tiller
spec:
  group: certmanager.k8s.io
  version: v1alpha1
  names:
    kind: ClusterIssuer
    plural: clusterissuers
  scope: Cluster

and I'm trying to create an object of its type:

apiVersion: certmanager.k8s.io/v1alpha1                                                                                                                                                                                                                                                                                                                                      
kind: ClusterIssuer
metadata:
  name: letsencrypt-issuer
  namespace: default
spec:
  ...

with

data "jsone_template" "certificate_clusterissuer" {
  template = "${file("${path.module}/certificate/clusterissuer.yaml")}"
}

resource "k8s_manifest" "certificate_clusterissuer" {
  content    = "${data.jsone_template.certificate_clusterissuer.rendered}"
}

(jsone_template in this case is equivalent to a yaml parser, as there are no JSON-e operators or expressions in the input)

Terraform apply creates the resource correctly:

tf@eb5a5de314f4:/repo/tf$ kubectl get --ignore-not-found clusterissuer/letsencrypt-issuer
NAME                 AGE
letsencrypt-issuer   18m

and

tf@eb5a5de314f4:/repo/tf$ terraform state show module.taskcluster.k8s_manifest.certificate_clusterissuer
id      = /apis/certmanager.k8s.io/v1alpha1/letsencrypt-issuer
content = {"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"ClusterIssuer","metadata":{"name":"letsencrypt-issuer","namespace":"default"},"spec":{"acme":{"email":"[email protected]","http01":{},"privateKeySecretRef":{"name":"letsencrypt-staging"},"server":"https://acme-staging-v02.api.letsencrypt.org/directory"}}}

but the resulting 'id' seems to cause issues on subsequent terraform runs:

* module.taskcluster.k8s_manifest.certificate_clusterissuer: k8s_manifest.certificate_clusterissuer: /google-cloud-sdk/bin/kubectl kubectl get --ignore-not-found v1alpha1/letsencrypt-issuer exit status 1: the server doesn't have a resource type "v1alpha1"

The selfLink for the resource is /apis/certmanager.k8s.io/v1alpha1/letsencrypt-issuer, so

terraform-provider-k8s/main.go

Lines 190 to 195 in 6a55723

	func resourceFromSelflink(s string) (resource, namespace string, ok bool) {
	parts := strings.Split(s, "/")
	if len(parts) < 2 {
	return "", "", false
	}
	resource = parts[len(parts)-2] + "/" + parts[len(parts)-1]

naturally produces that k8s resource name. I'm not sure where the bug is, here -- is there a spec for the format of the selfLink that is being violated here?

/cc @imbstack

handle manifests with multiple resources?

e2e tests

Support resources with ":" in their name

I was able to successfully deploy the manifests from https://github.com/kubernetes-incubator/metrics-server/tree/master/deploy/1.8%2B (after splitting out the files with multiple resources), but when I try to run a plan or apply, it is escaping the ":"s to "%3A"s which causes kubectl to fail:

* module.metrics_server.k8s_manifest.manifest: 3 error(s) occurred:

* module.metrics_server.k8s_manifest.manifest[2]: k8s_manifest.manifest.2: /home/pib/bin/kubectl kubectl --kubeconfig /tmp/kubeconfig_598509624 get --ignore-not-found clusterrolebindings/metrics-server%3Asystem%3Aauth-delegator exit status 1: invalid resource name "metrics-server%3Asystem%3Aauth-delegator": [may not contain '%']

* module.metrics_server.k8s_manifest.manifest[0]: k8s_manifest.manifest.0: /home/pib/bin/kubectl kubectl --kubeconfig /tmp/kubeconfig_136935660 get --ignore-not-found clusterroles/system%3Ametrics-server exit status 1: invalid resource name "system%3Ametrics-server": [may not contain '%']

* module.metrics_server.k8s_manifest.manifest[1]: k8s_manifest.manifest.1: /home/pib/bin/kubectl kubectl --kubeconfig /tmp/kubeconfig_072056062 get --ignore-not-found clusterrolebindings/system%3Ametrics-server exit status 1: invalid resource name "system%3Ametrics-server": [may not contain '%']