The Kubernetes StackSet Controller is a concept (along with an implementation) for easing and automating application life cycle for certain types of applications running on Kubernetes.
It is not meant to be a generic solution for all types of applications but it's explicitly focusing on "Web Applications", that is, application which receive HTTP traffic and are continuously deployed with new versions which should receive traffic either instantly or gradually fading traffic from one version of the application to the next one. Think Blue/Green deployments as one example.
By default Kubernetes offers the Deployment resource type which, combined with a Service, can provide some level of application life cycle in the form of rolling updates. While rolling updates are a powerful concept, there are some limitations for certain use cases:
- Switching traffic in a Blue/Green style is not possible with rolling updates.
- Splitting traffic between versions of the application can only be done by scaling the number of Pods. E.g. if you want to give 1% of traffic to a new version, you need at least 100 Pods.
- Impossible to run smoke tests against a new version of the application before it gets traffic.
To work around these limitations I propose a different type of resource called
an StackSet which has the concept of Stacks.
The StackSet is a declarative way of describing the application stack as a
whole, and the Stacks describe individual versions of the
application. The StackSet also allows defining a "global" load balancer
spanning all stacks of the stackset which makes it possible to switch
traffic to different stacks at the load balancer (for example Ingress) level.
flowchart
A[Ingress Controller] -->|0%| B[Stack\nVersion 1]
A[Ingress Controller] -->|20%| C[Stack\nVersion 2]
A[Ingress Controller] -->|80%| D[Stack\nVersion 3]
The StackSet and Stack resources are implemented as
CRDs. A StackSet looks like this:
apiVersion: zalando.org/v1
kind: StackSet
metadata:
name: my-app
spec:
# optional Ingress definition.
ingress:
hosts: [my-app.example.org, alt.name.org]
backendPort: 80
# optional desired traffic weights defined by stack
traffic:
- stackName: mystack-v1
weight: 80
- stackName: mystack-v2
weight: 20
# optional percentage of required Replicas ready to allow traffic switch
# if none specified, defaults to 100
minReadyPercent: 90
stackLifecycle:
scaledownTTLSeconds: 300
limit: 5 # maximum number of scaled down stacks to keep.
# If there are more than `limit` stacks, the oldest stacks which are scaled down
# will be deleted.
stackTemplate:
spec:
version: v1 # version of the Stack.
replicas: 3
# optional autoscaler definition (will create an HPA for the stack).
autoscaler:
minReplicas: 3
maxReplicas: 10
metrics:
- type: CPU
averageUtilization: 50
# full Pod template.
podTemplate:
spec:
containers:
- name: skipper
image: ghcr.io/zalando/skipper:latest
args:
- skipper
- -inline-routes
- '* -> inlineContent("OK") -> <shunt>'
- -address=:80
ports:
- containerPort: 80
name: ingress
resources:
limits:
cpu: 10m
memory: 50Mi
requests:
cpu: 10m
memory: 50MiThe above StackSet would generate a Stack that looks like this:
apiVersion: zalando.org/v1
kind: Stack
metadata:
name: my-app-v1
labels:
stackset: my-app
stackset-version: v1
spec:
ingress:
hosts: [my-app.example.org, alt.name.org]
backendPort: 80
replicas: 3
autoscaler:
minReplicas: 3
maxReplicas: 10
metrics:
- type: CPU
averageUtilization: 50
podTemplate:
spec:
containers:
image: ghcr.io/zalando/skipper:latest
args:
- skipper
- -inline-routes
- '* -> inlineContent("OK") -> <shunt>'
- -address=:80
ports:
- containerPort: 80
name: ingress
resources:
limits:
cpu: 10m
memory: 50Mi
requests:
cpu: 10m
memory: 50MiFor each Stack, the StackSet controller creates a Service, a Deployment,
and an optional Ingress resource automatically with the right labels. An
optional autoscaler resource can also be created per stack for
horizontally scaling the deployment.
For the most part the Stacks will be dynamically managed by the
controller and the users don't have to touch them. You can think of this similar
to the relationship between Deployments and ReplicaSets.
If the Stack is deleted the related resources like Service and
Deployment will be automatically cleaned up.
The stackLifecycle let's you configure two settings to change the cleanup
behavior for the StackSet:
scaleDownTTLSecondsdefines for how many seconds a stack should not receive traffic before it's scaled down.limitdefines the total number of stacks to keep. That is, if you have alimitof5and currently have6stacks for theStackSetthen it will clean up the oldest stack which is NOT getting traffic. Thelimitis not enforced if it would mean deleting a stack with traffic. E.g. if you set alimitof1and have two stacks with50%then none of them would be deleted. However, if you switch to100%traffic for one of the stacks then the other will be deleted after it has not received traffic forscaleDownTTLSeconds.
- Automatically create new Stacks when the
StackSetis updated with a new version in thestackTemplate. - Traffic switch between Stacks: The controller creates a new Ingress and/or
RouteGroup per Stack for StackSets with a
routegrouporingressspecified in thespec. The controller automatically updates each Stacks' Ingress/RouteGroup when updating the main StackSet'strafficweights. The ingress controller must implement theTrafficSegmentpredicate to effectively switch traffic. For example, Skipper implements this predicate. - Safely switch traffic to scaled down stacks. If a stack is scaled down, it will be scaled up automatically before traffic is directed to it.
- Dynamically provision Ingresses per stack, with per stack host names. I.e.
my-app.example.org,my-app-v1.example.org,my-app-v2.example.org. - Automatically scale down stacks when they don't get traffic for a specified period.
- Automatically delete stacks that have been scaled down and are not getting any traffic for longer time.
- Automatically clean up all dependent resources when a
StackSetorStackresource is deleted. This includesService,Deployment,Ingressand optionallyHorizontalPodAutoscaler. - Command line utility (
traffic) for showing and switching traffic between stacks. - You can opt-out of the global
Ingresscreation withexternalIngress:spec, such that external controllers can manage the Ingress or CRD creation, that will configure the routing into the cluster. - You can use skipper's RouteGroups to configure more complex routing rules.
The StackSet controller works with Kubernetes >=v1.23.
The controller watches for StackSet resources and creates Stack resources
whenever the version is updated in the StackSet stackTemplate. For each
StackSet it will create an optional "main" Ingress resource and keep it up
to date when new Stacks are created for the StackSet. For each Stack it
will create a Deployment, a Service. When specified in the parent
StackSet, the controller will create also for each Stack an Ingress and/or
a HorizontalPodAutoscaler for the Deployment. The corresponding Stack owns
these resources, which are cleaned up if the stack is deleted.
Use an existing cluster or create a test cluster with kind
kind create cluster --name testcluster001The stackset-controller can be run as a deployment in the cluster.
See deployment.yaml.
The controller depends on the StackSet and Stack CRDs. You must install these into your cluster before running the controller:
$ kubectl apply -f docs/stackset_crd.yaml -f docs/stack_crd.yamlAfter the CRDs are installed the controller can be deployed:
please adjust the controller version and cluster-domain to your environment
$ kubectl apply -f docs/rbac.yaml -f docs/deployment.yamlThere are cases where it might be desirable to run multiple instances of the stackset-controller in the same cluster, e.g. for development.
To prevent the controllers from fighting over the same StackSet resources
they can be configured with the flag --controller-id=<some-id> which
indicates that the controller should only manage the StackSets which has an
annotation stackset-controller.zalando.org/controller=<some-id> defined.
If the controller-id is not configured, the controller will manage all
StackSets which does not have the annotation defined.
Once you have deployed the controller you can create your first StackSet
resource:
$ kubectl apply -f docs/stackset.yaml
stackset.zalando.org/my-app createdThis will create the stackset in the cluster:
$ kubectl get stacksets
NAME CREATED AT
my-app 21sAnd soon after you will see the first Stack of the my-app
stackset:
$ kubectl get stacks
NAME CREATED AT
my-app-v1 30sIt will also create Ingress, Service, Deployment and
HorizontalPodAutoscaler resources:
$ kubectl get ingress,service,deployment.apps,hpa -l stackset=my-app
NAME HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/my-app-v1-traffic-segment my-app.example.org kube-ing-lb-3es9a....elb.amazonaws.com 80 7m
ingress.networking.k8s.io/my-app-v1 my-app-v1.example.org kube-ing-lb-3es9a....elb.amazonaws.com 80 7m
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/my-app-v1 ClusterIP 10.3.204.136 <none> 80/TCP 7m
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.apps/my-app-v1 1 1 1 1 7m
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/my-app-v1 Deployment/my-app-v1 <unknown>/50% 3 10 0 20sImagine you want to roll out a new version of your stackset. You can do this
by changing the StackSet resource. E.g. by changing the version:
$ kubectl patch stackset my-app --type='json' -p='[{"op": "replace", "path": "/spec/stackTemplate/spec/version", "value": "v2"}]'
stackset.zalando.org/my-app patchedSoon after, we will see a new stack:
$ kubectl get stacks -l stackset=my-app
NAME CREATED AT
my-app-v1 14m
my-app-v2 46sAnd using the traffic tool we can see how the traffic is distributed (see
below for how to build the tool):
./build/traffic my-app
STACK TRAFFIC WEIGHT
my-app-v1 100.0%
my-app-v2 0.0%If we want to switch 100% traffic to the new stack we can do it like this:
# traffic <stackset> <stack> <traffic>
./build/traffic my-app my-app-v2 100
STACK TRAFFIC WEIGHT
my-app-v1 0.0%
my-app-v2 100.0%Since the my-app-v1 stack is no longer getting traffic it will be scaled down
after some time and eventually deleted.
If you want to delete it manually, you can simply do:
$ kubectl delete stack my-app-v1
stack.zalando.org "my-app-v1" deletedAnd all the related resources will be gone shortly after:
$ kubectl get ingress,service,deployment.apps,hpa -l stackset=my-app,stack-version=v1
No resources found.This project uses Go modules as introduced in Go 1.11 therefore you need Go >=1.11 installed in order to build. If using Go 1.11 you also need to activate Module support.
Assuming Go has been setup with module support it can be built simply by running:
$ export GO111MODULE=on # needed if the project is checked out in your $GOPATH.
$ makeNote that the Go client interface for talking to the custom StackSet and
Stack CRD is generated code living in pkg/client/ and
pkg/apis/zalando.org/v1/zz_generated_deepcopy.go. If you make changes to
pkg/apis/* then you must run make clean && make to regenerate the code.
To understand how this works see the upstream example for generating client interface code for CRDs.
Clients that write the desired traffic switching value have to move
from ingress annotation zalando.org/stack-traffic-weights: '{"mystack-v1":80, "mystack-v2": 20}'
to stackset spec.traffic:
spec:
traffic:
- stackName: mystack-v1
weight: 80
- stackName: mystack-v2
weight: 20
![dependabot-preview[bot] avatar](https://avatars.githubusercontent.com/in/2141?v=4 "dependabot-preview[bot]")
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent