ahmetb/gke-letsencrypt

Tutorial for installing cert-manager on GKE get HTTPS certificates from Let’s Encrypt (⚠️NOW OBSOLETE⚠️)

💎🔥💎🔥💎

Important Update (April 9,2019)

GKE now has beta support for natively provisioning Let's Encrypt TLS certificates.
This means you should stop using this tutorial and use the feature provided by GKE instead.
💎🔥💎🔥💎

Let’s Encrypt on GKE

GKE (Google Kubernetes Engine) does not offer an out-of-the-box HTTPS solution or TLS/SSL certificates for your websites today:

• Let’s Encrypt is a non-profit Certificate Authority that provides free TLS/SSL certificates that can be used to secure websites with HTTPS.
• cert-manager is a third-party Kubernetes controller that automates getting TLS/SSL certificates from Let’s Encrypt and refreshing them.

⚠️⚠️ cert-manager is pre-stable software and is not officially supported by Google. Use it at your own risk! ⚠️⚠️

Requirements:

• A registered domain name
• A GKE cluster
• Estimated time: 30 minutes.

Steps

1. Install Helm
2. Install cert-manager
3. Set up Let's Encrypt
4. Deploy a web app on a domain name
5. Get a certificate for your domain name
6. Cleanup

---

What's not covered in this tutorial

• Redirecting HTTP traffic to HTTPS (not possible with GKE Ingress yet)
• Securing traffic between Cloud Load Balancer and your app with TLS

Alternative HTTPS proxies

If you're looking for a far simpler third-party solution and you're OK with HTTPS requests from your visitors terminated/proxied by a third-party, these services work with GKE apps:

• Cloudflare (has a free tier)
• Backplane

---

This is not an official Google product or documentation.

gke-letsencrypt's Issues

Cert-manager support automatic assigments of certificates and it should be preffered

https://github.com/jetstack/cert-manager/blob/master/docs/reference/ingress-shim.rst

Nginx Ingress example?

Any chance of additional example with Nginx ingress?
Seems a lot of people are using it due to being a superior solution, it might be good idea..

Installing cert manager throws error

Error: namespaces "kube-system" is forbidden: User "system:serviceaccount:kube-system:default" cannot get namespaces in the namespace "kube-system"

I have followed all the instructions up to this point. I have ran them twice and gotten confirmation that they were properly installed.

Error on GKE

I have followed the guide down to the T and this is the error showing up on the console on GCP:

Could not find TLS certificates. Continuing setup for the load balancer to serve HTTP. Note: this behavior is deprecated and will be removed in a future version of ingress-gce

Doing a kubectl describe certificate shows that the certificate has been successfully issued and going to the https on my website seems also fine. So wondering why that error is showing up.

Also login into the admin pages of the website no longer works. Any pointers or help?

cert-manager v0.5.2 install broken on Helm v2.12.0

Ran through this today after previous success (well, previous cert-manager bugs notwithstanding), and got the following error: cert-manager/cert-manager#1144

error while evaluating the ingress spec: could not find service "default/cm-acme-http-solver-cjnqk"

The certificate gets issued

Conditions:
    Last Transition Time:  2018-12-16T02:35:38Z
    Message:               Certificate issued successfully

However, the load balancer only gets a HTTP/80 frontend and has the backend in UNKNOWN state.

DNS01 method?

Great guide Ahmet, any chance of providing additional instructions or modifying current ones to use DNS method? Seems a bit less error-prone and more manageable in a long run.

Thanks in advance!

Tracking pixels

What's the deal with the tracking pixels in the markdown files? To my knowledge that's not legal in many jurisdictions without an accompanying privacy policy, which I'm not seeing. Besides, is it even effective given GitHub's Camo image proxy?

GitHub already has a traffic page in case you weren't aware, which is obviously covered by their privacy policy.

Issuer.yaml out of date

Cheers for a great walkthrough, saved me a ton of headscratching.

Acme server URL needs updating in issuer.yaml - otherwise the Certificate always hangs with letsencrypt-prod not ready. On describe the error is:

...
   Message:               Your ACME server URL is set to a v1 endpoint (https://acme-v01.api.letsencrypt.org/directory). You should update the spec.acme.server field to "https://acme-v02.api.letsencrypt.org/directory"
    Reason:                InvalidConfig
    Status:                False
    Type:                  Ready
...

Easily fixed by updating issuer.yaml:

...
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
...

...
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
...

TLS ingress provisioning automation?

Great guide!
Is there a way to automate provisioning of the TLS ingress once ACME validation is over, to avoid applying ingress yaml twice?

Thanks!

Check renewal

Hi,
Thanks, your tutorial worked like a charm on GKE !
However, in the section "Get a certificate for your domain" , I didn't get the
"Normal RenewalScheduled Certificate scheduled for renewal in 1438 hours" line (see [1]).
How to check if my certificates will be renewed as expected ?

Thanks

Events:
  Type    Reason          Age   From          Message
  ----    ------          ----  ----          -------
  Normal  CreateOrder     44m   cert-manager  Created new ACME order, attempting validation...
  Normal  DomainVerified  37m   cert-manager  Domain "dogs.test.florat.net" verified with "http-01" validation
  Normal  IssueCert       37m   cert-manager  Issuing certificate...
  Normal  CertObtained    37m   cert-manager  Obtained certificate from ACME server
  Normal  CertIssued      37m   cert-manager  Certificate issued successfully

[Question] Manage list of domains

Hello,

I'm on project with multiple domains, and some domains can be added whenever.
I want to know, how factoring to have one list of domains ? ( And change list + deploy to have the good certificate)
Config values ? secret ? Helm variables ?
Make on certificate with all domains or a certificate by domain ?

Another question : we need the staging config ?

So thanks for this tutorial, that's very clear !

Not working (secret is not created)

Followed the tutorial step by step.

Never got this

Type     Reason                Message
----     ------                -------
Warning  ErrorCheckCertificate Error checking existing TLS certificate: secret "www-dogs-com-tls" not found
Normal   PrepareCertificate    Preparing certificate with issuer
Normal   PresentChallenge      Presenting http-01 challenge for domain foo.kubernetes.tips
Normal   SelfCheck             Performing self-check for domain www.dogs.com
Normal   ObtainAuthorization   Obtained authorization for domain www.dogs.com
Normal   IssueCertificate      Issuing certificate...
Normal   CeritifcateIssued     Certificated issued successfully
Normal   RenewalScheduled      Certificate scheduled for renewal in 1438 hours

Got this instead

  Type    Reason          Age   From          Message
  ----    ------          ----  ----          -------
  Normal  CreateOrder     27m   cert-manager  Created new ACME order, attempting validation...
  Normal  DomainVerified  20m   cert-manager  Domain "ingresstest.myowndomain.com" verified with "http-01" validation

Which seems fine to me but the secret never got populated kubectl get secrets

And the domain never got the SSL certificate.

Will keep on investigating and will update the tutorial if I find a solution.

Error: namespaces "kube-system"

Maybe I missed something here, but I'm pretty sure I came further before but retried this tutorial now with 0.5.0, before was 0.4.1 I think:

~/d/p/kb-ops ❯❯❯ helm install --name cert-manager --version v0.5.0 \                                                                                                                                  master ✱ ◼
    --namespace kube-system stable/cert-manager
Error: namespaces "kube-system" is forbidden: User "system:serviceaccount:kube-system:tiller" cannot get namespaces in the namespace "kube-system": Unknown user "system:serviceaccount:kube-system:tiller"

Give more document space to the DNS setup step

LOVING this tutorial so far, but I noticed one issue on this page:

https://github.com/ahmetb/gke-letsencrypt/blob/master/40-deploy-an-app.md

After it works, update your domain name records (at your domain registrar or DNS provider) with this IP address.

This is a critical instruction, but you presented it as a single line with no highlight or bullet or anything. It's easy to gloss over instructions like this. I understand not wanting to link to every conceivable DNS update mechanism, but can you just make this take up more space on the page? Even something as simple as adding a bullet or some bold would help it stand out. (Notably, the other instructions on the same page don't need anything extra, because they're followed by code blocks.)

Great job on this tutorial in general, I was really struggling with cert-manager until I found it!

`.well-known/acme-challenge/...` endpoint causes warning after certificate issued

The endpoint .well-known/acme-challenge/... causes a warning, as the cert-manager service it points to is removed after the first certificate is issued.

Does the .well-known/acme-challenge/... endpoint need to stay configured, or can it be removed after the certificate is issued? Is it just used once for proof of ownership, or is needed for certificate renewal.

Given it causes the warning, it would be good to clarify this in the walkthrough.

Convert from staging to prod

I was able to successfully request a cert using the letsencrypt-staging issuer. Since it worked, I want to switch it to prod. I updated the yaml and reapplied it. If I describe it, I see that it is using the new yaml with letsencrypt-prod.

But how/when does it go out and update the cert to to the prod version?

Thanks!

Error with `helm init --service-account tiller`

When I run helm init --service-account tiller I get the following:

Warning: Tiller is already installed in the cluster.
(Use --client-only to suppress this message, or --upgrade to upgrade Tiller to the current version.)
Happy Helming!

If I continue to try to install the cert-manager then I get this issue #25

I already installed Tiller because it was the next step right after installing Helm on the page that you linked to in the instructions.

Webhook Error

When I run this command:

curl -sSL https://rawgit.com/ahmetb/gke-letsencrypt/master/yaml/letsencrypt-issuer.yaml | \
>     sed -e "s/email: ''/email: $EMAIL/g" | \
>     kubectl apply -f-

I receive this error:

Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling admission webhook "clusterissuers.admission.certmanager.k8s.io": the server is currently unable to handle the request
Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling admission webhook "clusterissuers.admission.certmanager.k8s.io": the server is currently unable to handle the request

I am trying to upgrade to the cert manager v0.5.2 process.

I found this issue, but the solutions did not work.
https://docs.cert-manager.io/en/latest/getting-started/webhook.html

Error when installing LetsEncrypt

Any idea on the following error?

Error: release cert-manager failed: clusterroles.rbac.authorization.k8s.io "cert-manager-cert-manager" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["certificates"], APIGroups:["certmanager.k8s.io"], Verbs:["*"]} PolicyRule{Resources:["issuers"], APIGroups:["certmanager.k8s.io"], Verbs:["*"]} PolicyRule{Resources:["clusterissuers"], APIGroups:["certmanager.k8s.io"], Verbs:["*"]} PolicyRule{Resources:["secrets"], APIGroups:[""], Verbs:["*"]} PolicyRule{Resources:["events"], APIGroups:[""], Verbs:["*"]} PolicyRule{Resources:["endpoints"], APIGroups:[""], Verbs:["*"]} PolicyRule{Resources:["services"], APIGroups:[""], Verbs:["*"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["*"]} PolicyRule{Resources:["ingresses"], APIGroups:["extensions"], Verbs:["*"]}] user=&{system:serviceaccount:kube-system:default 46139d93-0134-11e8-854c-42010a9a0fc6 [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]} ownerrules=[PolicyRule{Resources:["selfsubjectaccessreviews"], APIGroups:["authorization.k8s.io"], Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]} PolicyRule{NonResourceURLs:["/swagger-2.0.0.pb-v1"], Verbs:["get"]} PolicyRule{NonResourceURLs:["/swagger.json"], Verbs:["get"]}] ruleResolutionErrors=[]

Certificate is not issued

step 50
cert-manager-v0.6.6

$ kubectl describe certificate singh-secret
Name:         singh-secret
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2019-03-13T10:42:40Z
  Generation:          1
  Owner References:
    API Version:           extensions/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  helloweb
    UID:                   b999fbbe-457c-11e9-a2d5-42010a9400fd
  Resource Version:        2042233
  Self Link:               /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/singh-secret
  UID:                     b9b3bc9b-457c-11e9-a2d5-42010a9400fd
Spec:
  Acme:
    Config:
      Domains:
        singh.hbot.dev
      Http 01:
        Ingress:  helloweb
  Dns Names:
    singh.hbot.dev
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt-prod
  Secret Name:  singh-secret
Status:
  Conditions:
    Last Transition Time:  2019-03-13T10:42:40Z
    Message:               Certificate does not exist
    Reason:                NotFound
    Status:                False
    Type:                  Ready
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Generated  56m   cert-manager  Generated new private key

After wait for almost hour. No progress.

http->https redirect?

Not an issue per se, but it doesn't appear that the annotation for the regular (GCE)/ingress controller works in this case (using: kubernetes.io/ingress.allow-http: "false").

Do I need to upgrade to the Nginx controller? Can that be done in the step before requesting the cert such that the automation to update the ingress manifest will work?

https://github.com/kubernetes/ingress-nginx/

http-01 self check failed for domain

I am trying to create a Let's encrypt certificate for my domain using Ingress.

Unfortunately my cert doesn't get generated.

kubectl describe certificate

Name:         project-ingress-tls
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2019-02-22T08:57:58Z
  Generation:          1
  Owner References:
    API Version:           extensions/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  project-ingress
    UID:                   21d94d2e-35b9-11e9-b101-42010a9a003c
  Resource Version:        231826
  Self Link:               /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/project-ingress-tls
  UID:                     f35a2b83-367f-11e9-b6e9-42010a9a01da
Spec:
  Acme:
    Config:
      Domains:
        kube.project.net
      Http 01:
        Ingress:  project-ingress
  Dns Names:
    kube.project.net
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt-prod
  Secret Name:  project-ingress-tls
Status:
  Acme:
    Order:
      Challenges:
        Authz URL:  https://acme-v02.api.letsencrypt.org/acme/authz/dNE88BRLPQL3k9E_rTEoDKYSP2lyNT1hrjYu2Ca71q8
        Domain:     kube.project.net
        Http 01:
          Ingress:  project-ingress
        Key:        key.goes-here
        Token:      token
        Type:       http-01
        URL:        https://acme-v02.api.letsencrypt.org/acme/challenge/dNE88BRLPQL3k9E_rTEoDKYSP2lyNT1hrjYu2Ca71q8/12862928805
        Wildcard:   false
      URL:          https://acme-v02.api.letsencrypt.org/acme/order/51999635/326346013
  Conditions:
    Last Transition Time:  2019-02-22T09:13:51Z
    Message:               http-01 self check failed for domain "kube.project.net"
    Reason:                ValidateError
    Status:                False
    Type:                  Ready
Events:
  Type    Reason       Age   From          Message
  ----    ------       ----  ----          -------
  Normal  CreateOrder  16m   cert-manager  Created new ACME order, attempting validation...

Service gets created

cm-acme-http-solver-bbdrn   NodePort    10.23.242.207   <none>        8089:30685/TCP            9m

Ingress was modified, kubectl get ingress -o=yaml project-ingress

- backend:
          serviceName: cm-acme-http-solver-bbdrn
          servicePort: 8089
        path: /.well-known/acme-challenge/EfrlcexUdaMdl8QUuIJhpJt3bGCF2teBpmKZpdhCGRI

status:
  loadBalancer:
    ingress:
    - ip: 35.189.115.86

Actually when I've tried to reach the: 35.189.115.86:8089 or 35.189.115.86:30685 from the outside I get the timeout.
My ingress doesn't use the LoadBalancer by default, it was added by this cert manager.
The IP: 35.189.115.86 is my Kubernetes node IP, so it doesn't look like load balancer was created correctly. If that's the case I shall probably whitelist the firewall? gcloud compute firewall-rules create myservice --allow tcp:8089

Any idea what might be the problem? Thanks

HTTPS call is not working

I followed this tutorial, but the last step is not working (serving through HTTPS).
The secret is created, I can access it on HTTP but not on HTTPS.
Can you guide me to the fix?

this is the output of the describe ingress:

Name: helloweb
Namespace: default
Address: 35.190.68.173
Default backend: helloweb-backend:8080 (10.32.0.17:8080)
TLS:
dogs-com-tls terminates app-solidair-vlaanderen.com
Rules:
Host Path Backends

---

app-solidair-vlaanderen.com
/.well-known/acme-challenge/Q8kcFSZ0ZUJO58xZyVbK6s-cJIWu-EgwPcDd8NFyoXQ cm-acme-http-solver-mhqnf:8089 ()
Annotations:
url-map: k8s-um-default-helloweb--17a833239f9491d9
backends: {"k8s-be-30819--17a833239f9491d9":"Unknown","k8s-be-32482--17a833239f9491d9":"HEALTHY"}
forwarding-rule: k8s-fw-default-helloweb--17a833239f9491d9
target-proxy: k8s-tp-default-helloweb--17a833239f9491d9
Events:
Type Reason Age From Message

---

Normal ADD 45m loadbalancer-controller default/helloweb
Normal CREATE 44m loadbalancer-controller ip: 35.190.68.173
Warning Sync 7m (x22 over 28m) loadbalancer-controller Error during sync: error while evaluating the ingress spec: could not find service "default/cm-acme-http-solver-mhqnf"

Cert Renewal

Hi - I'm wondering what is the recommended way to renew certs? The tutorial was great and worked, but 2 months later (now) I got a notification that my certificate will expire in a month. How do you recommend I renew the certificate? I'd prefer not to re-route my domain name to the cert solver again, as this will block access to my app and interrupt my customers.

GKE with Cloud Endpoints (Help)

First off, thank you for such a great guide. I am stuck with one aspect. I have a letsencrypt-prod cluster BUT I am using cloud endpoints (not an ingress controller). How should I point letsencrypt to the Google Cloud Load Balancer?

Service / Deployment

# kubectl create -f config.yaml
apiVersion: v1
kind: Service
metadata:
  name: tensortask-api
spec:
  ports:
  - port: 80
    targetPort: 9000
    protocol: TCP
    name: http1
  - port: 81
    targetPort: 9001
    protocol: TCP
    name: http2
  selector:
    app: tensortask-api
  type: LoadBalancer
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: tensortask-api
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: tensortask-api
    spec:
      containers:
      - name: esp
        image: gcr.io/endpoints-release/endpoints-runtime:1
        args: [
          "--http_port", "9000",  # for HTTP 1.1
          "--http2_port", "9001",  # for gRPC
          "-a", "grpc://127.0.0.1:8000",
          "-s", "api.tensortask.com",
          "--rollout_strategy", "managed",
          ]
        ports:
          - containerPort: 9000
          - containerPort: 9001

      - name: go-backend
        image: gcr.io/tensortask/server:latest
        ports:
          - containerPort: 8000

Certificate

apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
  name: tensortask-com-tls
  namespace: default
spec:
  secretName: tensortask-com-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: api.tensortask.com
  dnsNames:
  - api.tensortask.com
  acme:
    config:
    - http01:
        ingress: ??????????
      domains:
      - api.tensortask.com