suse / kucero Goto Github PK

KUbernetes CErtificate ROtation

License: Apache License 2.0

Makefile 1.95% Go 92.04% Dockerfile 0.38% Shell 5.62%

kucero's Introduction

Introduction

Kucero (KUbernetes CErtificate ROtation) is a Kubernetes daemonset that performs automatic Kubernetes control plane certificate rotation.

Kucero takes care both:

kubeadm certificates and kubeconfigs: kucero periodically watches the kubeadm generated certificates and kubeconfigs on host system, and renews certificates/kubeconfigs when the certificates/kubeconfigs residual time is below than user configured time period.
kubelet certificates:
- kubelet.conf: kucero helps on auto-update the /etc/kubernetes/kubelet.conf from embedded base64 encoded client cert/key to using the local file /var/lib/kubelet/kubelet-client-current.pem (this is a bug if you bootstrap a cluster with kubeadm version < 1.17).
- client certificate: kucero helps on configuring rotateCertificates: true or rotateCertificates: false in /var/lib/kubelet/config.yaml which controls to auto rotates the kubelet client certificate or not. When configures rotateCertificates: true, the kubelet sends out the client CSR at approximately 70%-90% of the total lifetime of the certificate, then the kube-controler-manager watches kubelet client CSR, and then auto signs and approves kubelet client certificates with Kubernetes cluster CA cert/key pair.
- server certificate: kucero helps on configuring serverTLSBootstrap: true or serverTLSBootstrap: false in /var/lib/kubelet/config.yaml which controls to auto rotates the kubelet server certificate or not. When configures serverTLSBootstrap: true, the kubelet sends out the server CSR at approximately 70%-90% of the total lifetime of the certificate, then the kucero controller watches kubelet server CSR, and then auto signs and approves kubelet server certificates with user-specified CA cert/key pair.

Kubelet Configuration

By default, kucero enables kubelet client rotateCertificates: true and server certificates serverTLSBootstrap: true auto rotation, you could disable it by passing flags to kucero:

--enable-kubelet-client-cert-rotation=false
--enable-kubelet-server-cert-rotation=false

Build Requirements

Golang >= 1.17
Docker
Kustomize

Container Requirement Package

/usr/bin/nsenter

Kubeadm Compatibility

kubeadm >= 1.15.0

Installation

make docker-build IMG=<YOUR-DOCKER-REPOSITORY-IMAGE-NAME-TAG>
make docker-push IMG=<YOUR-DOCKER-REPOSITORY-IMAGE-NAME-TAG>
make deploy-manifest IMG=<YOUR-DOCKER-REPOSITORY-IMAGE-NAME-TAG>

Configuration

The following arguments can be passed to kucero via the daemonset pod template:

Flags:
      --ca-cert-path string         sign CSR with this certificate file (default "/etc/kubernetes/pki/ca.crt")
      --ca-key-path string          sign CSR with this private key file (default "/etc/kubernetes/pki/ca.key")
      --ds-name string              name of daemonset on which to place lock (default "kucero")
      --ds-namespace string         namespace containing daemonset on which to place lock (default "kube-system")
      --enable-kucero-controller    enable kucero controller (default true)
  -h, --help                        help for kucero
      --leader-election-id string   the name of the configmap used to coordinate leader election between kucero-controllers (default "kucero-leader-election")
      --lock-annotation string      annotation in which to record locking node (default "caasp.suse.com/kucero-node-lock")
      --metrics-addr string         the address the metric endpoint binds to (default ":8080")
      --polling-period duration     certificate rotation check period (default 1h0m0s)
      --renew-before duration       rotates certificate before expiry is below (default 720h0m0s)

Uninstallation

make destroy-manifest

Demo

kubeadm
kubelet

kucero's People

Contributors

Stargazers

Watchers

Forkers

vadorovsky tubbz-alt nallurichandu99 enj andrew-est nordix gabrielstein hoangph3 warmchang haiming2101

kucero's Issues

Remove dependency on kubectl binary

Use k8s.io/kubectl package instead.

Certificate authority (CA) rotation

Does this project help doing CA certificate rotation?

https://kubernetes.io/docs/tasks/tls/manual-rotation-of-ca-certificates/

Use KIND to do E2E test

Use KIND in Github action to do the E2E test.

Backdated go modules and a new release

The go modules are pretty backdated and as such susceptible to different vulnerabilities. I have opened a PR to fix this issue. After the PR is merged, lets have a new release since the last release was quite a log ago and is missing the last few important PRs.

Unsupported authorization API versions in kucero as per k8s 1.22 and approver adaptation issue

We tested kucero 1.5.1 CSRs signing in a kubernetes 1.22 cluster and noticed the CSRs were not being approved.

Kucero logs pointed to issues with the SAR. e.g:

time="2021-11-08T09:13:56Z" level=info msg=Signing
time="2021-11-08T09:13:56Z" level=error msg="SubjectAccessReview failed: the server could not find the requested resource"

We noticed that the Authorization API used in Kucero, is also V1beta1 and has been removed in Kubernetes 1.22.

The authorization.k8s.io/v1beta1 API version of LocalSubjectAccessReview, SelfSubjectAccessReview, and SubjectAccessReview is no longer served as of v1.22.

After changing the API to V1 and testing, the CSRs were issued but the state didn't got changed to approved. Checking the kubelet code because of this missing state, kubelet will not get the certificate.

We tried adding a condition in the approver part and that seems to work. Perhaps this was not needed in older versions.

The following changes lead to kucero working in k8s 1.22

diff --git a/controllers/certificatesigningrequest_controller.go b/controllers/certificatesigningrequest_controller.go
index 29144b4..c229d5d 100644
--- a/controllers/certificatesigningrequest_controller.go
+++ b/controllers/certificatesigningrequest_controller.go
@@ -21,7 +21,7 @@ import (
  "crypto/x509"
  "fmt"
- authorization "k8s.io/api/authorization/v1beta1"
+ authorization "k8s.io/api/authorization/v1"
  capi "k8s.io/api/certificates/v1"
  capiv1beta1 "k8s.io/api/certificates/v1beta1"
  corev1 "k8s.io/api/core/v1"
@@ -154,7 +154,7 @@ func (r *CertificateSigningRequestSigningReconciler) authorize(csr *capi.Certifi
      ResourceAttributes: &rattrs,
    },
  }
- sar, err := r.ClientSet.AuthorizationV1beta1().SubjectAccessReviews().Create(context.TODO(), sar, metav1.CreateOptions{})
+ sar, err := r.ClientSet.AuthorizationV1().SubjectAccessReviews().Create(context.TODO(), sar, metav1.CreateOptions{})
  if err != nil {
    return false, err
  }
@@ -165,6 +165,7 @@ func appendApprovalCondition(csr *capi.CertificateSigningRequest, message string
  csr.Status.Conditions = append(csr.Status.Conditions, capi.CertificateSigningRequestCondition{
    Type:    capi.CertificateApproved,
    Reason:  "AutoApproved by kucero",
+   Status:  corev1.ConditionTrue,
    Message: message,
  })
 }

Support auto-update kubelet.conf cert rotation if the cluster is bootstrapped by kubeadm < 1.17

If the admin uses kubeadm < 1.17 to bootstrap a cluster, the /etc/kubernetes/kubelet.conf embeds the client cert/key with base64 encoded, but not points to the auto rotates cert/key /var/lib/kubelet/pki/kubelet-client-current.pem.
To support an auto-update the /etc/kubernetes/kubelet.conf to use the symlink /var/lib/kubelet/pki/kubelet-client-current.pem, which is:

from

client-certificate-data: <base64-encoded-client-cert>
client-key-data: <base64-encoded-client-key>

client-certificate: /var/lib/kubelet/pki/kubelet-client-current.pem
client-key: /var/lib/kubelet/pki/kubelet-client-current.pem