ericchiang / terraform-provider-k8s Goto Github PK
View Code? Open in Web Editor NEWKubernetes Terraform provider with support for raw manifests
License: MIT License
Kubernetes Terraform provider with support for raw manifests
License: MIT License
I was able to successfully deploy the manifests from https://github.com/kubernetes-incubator/metrics-server/tree/master/deploy/1.8%2B (after splitting out the files with multiple resources), but when I try to run a plan or apply, it is escaping the ":"s to "%3A"s which causes kubectl to fail:
* module.metrics_server.k8s_manifest.manifest: 3 error(s) occurred:
* module.metrics_server.k8s_manifest.manifest[2]: k8s_manifest.manifest.2: /home/pib/bin/kubectl kubectl --kubeconfig /tmp/kubeconfig_598509624 get --ignore-not-found clusterrolebindings/metrics-server%3Asystem%3Aauth-delegator exit status 1: invalid resource name "metrics-server%3Asystem%3Aauth-delegator": [may not contain '%']
* module.metrics_server.k8s_manifest.manifest[0]: k8s_manifest.manifest.0: /home/pib/bin/kubectl kubectl --kubeconfig /tmp/kubeconfig_136935660 get --ignore-not-found clusterroles/system%3Ametrics-server exit status 1: invalid resource name "system%3Ametrics-server": [may not contain '%']
* module.metrics_server.k8s_manifest.manifest[1]: k8s_manifest.manifest.1: /home/pib/bin/kubectl kubectl --kubeconfig /tmp/kubeconfig_072056062 get --ignore-not-found clusterrolebindings/system%3Ametrics-server exit status 1: invalid resource name "system%3Ametrics-server": [may not contain '%']
It seems like right now there’s no specific configuration for the provider itself and it only defines a resource. It should allow to setup a kubectl context somewhere, because assuming that current context is the right one is pretty dangerous.
If it actually allows for that and I missed it please let me know.
I have a CRD:
# From https://github.com/jetstack/cert-manager/blob/912c7672bd3e6ba1e7ef9dff48083ee6f3d6cbab/contrib/manifests/cert-manager/with-rbac.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: clusterissuers.certmanager.k8s.io
annotations:
"helm.sh/hook": crd-install
labels:
app: cert-manager
chart: cert-manager-v0.6.0-dev.1
release: cert-manager
heritage: Tiller
spec:
group: certmanager.k8s.io
version: v1alpha1
names:
kind: ClusterIssuer
plural: clusterissuers
scope: Cluster
and I'm trying to create an object of its type:
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-issuer
namespace: default
spec:
...
with
data "jsone_template" "certificate_clusterissuer" {
template = "${file("${path.module}/certificate/clusterissuer.yaml")}"
}
resource "k8s_manifest" "certificate_clusterissuer" {
content = "${data.jsone_template.certificate_clusterissuer.rendered}"
}
(jsone_template in this case is equivalent to a yaml parser, as there are no JSON-e operators or expressions in the input)
Terraform apply creates the resource correctly:
tf@eb5a5de314f4:/repo/tf$ kubectl get --ignore-not-found clusterissuer/letsencrypt-issuer
NAME AGE
letsencrypt-issuer 18m
and
tf@eb5a5de314f4:/repo/tf$ terraform state show module.taskcluster.k8s_manifest.certificate_clusterissuer
id = /apis/certmanager.k8s.io/v1alpha1/letsencrypt-issuer
content = {"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"ClusterIssuer","metadata":{"name":"letsencrypt-issuer","namespace":"default"},"spec":{"acme":{"email":"[email protected]","http01":{},"privateKeySecretRef":{"name":"letsencrypt-staging"},"server":"https://acme-staging-v02.api.letsencrypt.org/directory"}}}
but the resulting 'id' seems to cause issues on subsequent terraform runs:
* module.taskcluster.k8s_manifest.certificate_clusterissuer: k8s_manifest.certificate_clusterissuer: /google-cloud-sdk/bin/kubectl kubectl get --ignore-not-found v1alpha1/letsencrypt-issuer exit status 1: the server doesn't have a resource type "v1alpha1"
The selfLink for the resource is /apis/certmanager.k8s.io/v1alpha1/letsencrypt-issuer
, so
terraform-provider-k8s/main.go
Lines 190 to 195 in 6a55723
/cc @imbstack
Hi, for people like me we are not Go developper and have no access to a Go development environment, can you make a release that we can download in GitHub?
Thanks
This is in follow up to PR #9.
I work at SlingTV. I am on an 8 person DevOps team. We support our developers, which at my estimation amount to something like 250+ engineers. We also support our ops team, who are responsible for the deployment of our backend software.
Our app deployment is large, and relies heavily on VMware, SaltStack, SSH, and even some manual deployment steps currently.
We are also in the early stages of putting our deployments into kubernetes. We already have several apps in k8s.
I would like to bring a large portion of our deployments under management by Terraform. However, if I can't import what is already deployed using old or undesirable technologies into Terraform state without affecting already-live deployments, putting stuff into TF is going to be a hard sell to my ops team. Likewise, if I can't re-create the TF state (if it were ever lost) without the need to touch current deployments, my Ops team probably won't let me introduce a new, never-before-used tool into their environment.
Fortunately, terraform import
allows for both use cases as described above. It exactly fits what I need to make sure that I can convince them to use terraform in our broader deployment strategy.
The only plugin that is even close to handling kubernetes deloyments right is, in my opinion, this one. It allows me to use CRDs and I don't have to wait for the the official kubernetes provider to support the workloads API.
I would like terraform import
to be supported with this plugin. This way I can make sure I can recover if terraform state were ever deleted without actually touching any live deployments. This will also allow me to write new terraform states, then import what I already have deployed into terraform, again without touching the live deploy.
I would use terraform import like this:
terraform import k8s_manifest.nginx-deployment /apis/extensions/v1beta1/namespaces/default/deployments/releaseme-example
That is, I would give terraform an ID of a live deployment and point to what terraform resource stanza I had written to tell TF what code should manage the live deploy in the future.
All that is required for that to work is to have the provider support what is known as an "id-only refresh", meaning that it has to be able to repopulate all of the attributes of the resource in its resourceManifestRead
function. So, if we can change that function to repopulate "content" based on what is actually already deployed, importing will work simply by adding some boilerplate just-use-the-read-function passthrough code when creating the Resource
struct and passing to terraform in the provider.
This is the approach I took in #9 . I simply read in from a k8s annotation what was previously given to kubectl in a kubectl apply
call. I also normalized the contents of the "contents" value so that diffs between what terraform reads in production and what terraform was given would make sense, and wouldn't just be over something like whitespace or quotes.
Are there any thoughts/plans on using the API [1] directly rather than via kubectl (or rely on locally persisted config/credentials)? This article [2] is interesting in this context.
[1] https://kubernetes.io/docs/concepts/overview/kubernetes-api/
[2] http://k8s.uk/kubectl-vs-http-api.html
First off, thanks for your work on this project, it's far more useful than the default k8s terraform provider.
Anyway, I'm curious why you chose to shell out to kubectl
versus using the kubernetes REST API. You describe why a bit in the README, but I'm curious why you can't use raw manifests w the REST API in a similar fashion? What's the advantage of working with "any Kubernetes resource natively"?
Changing the name, namespace or kind in the content
causes the provider to get confused about the resource that it is tracking. The read operations will continue to use the old names, etc... as these are encoded in the id associated with the resource. The creation operations, however, will use a combination of the newer values encoded in the content
and the older values encoded in the id.
Ultimately I think the name, namespace and kind should all be part of the terraform resource so that they can be ForceNew
. This avoids all of these strange edge cases.
I have a proof of concept implementation here and can get it into an MR if there is interest. It will, however, potentially conflict with #21
In the readme, it says to add this block to a ~/.terraformrc
providers {
k8s = "/$GOPATH/bin/terraform-provider-k8s"
}
I'm curious about that leading slash, is it intentional or a mistake? I did my best to read through the Terraform configuration syntax docs to find out whether it's special syntax for interpolating environment variables, but couldn't find anything.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.