stefanprodan/gitops-istio

A GitOps recipe for Progressive Delivery with Flux v2, Flagger and Istio

gitops-istio

This is a guide where you will get hands-on experience with GitOps and Progressive Delivery using Kubernetes and Istio.

Introduction

What is GitOps?

GitOps is a way to do Continuous Delivery, it works by using Git as a source of truth for declarative infrastructure and workloads. For Kubernetes this means using git push instead of kubectl apply/delete or helm install/upgrade.

In this workshop you'll be using GitHub to host the config repository and Flux as the GitOps delivery solution.

What is Progressive Delivery?

Progressive delivery is an umbrella term for advanced deployment patterns like canaries, feature flags and A/B testing. Progressive delivery techniques are used to reduce the risk of introducing a new software version in production by giving app developers and SRE teams a fine-grained control over the blast radius.

In this workshop you'll be using Flagger, Istio and Prometheus to automate Canary Releases and A/B Testing for your applications.

Prerequisites

You'll need a Kubernetes cluster v1.20 or newer with LoadBalancer support.

For testing purposes you can use Minikube with 2 CPUs and 4GB of memory:

minikube start --cpus='2' --memory='4g'

If using Minikube, run the following in a separate terminal window/tab for the duration of this workshop:

minikube tunnel

This assigns an External-IP to the istio-gateway service and allows the helm install to complete successfully.

Install jq, yq and the flux CLI with Homebrew:

brew install jq yq fluxcd/tap/flux

Verify that your cluster satisfies the prerequisites with:

flux check --pre

Fork this repository and clone it:

git clone https://github.com/<YOUR-USERNAME>/gitops-istio
cd gitops-istio

Cluster bootstrap

With flux bootstrap command you can install Flux on a Kubernetes cluster and configure it to manage itself from a Git repository. If the Flux components are present on the cluster, the bootstrap command will perform an upgrade if needed.

Bootstrap Flux by specifying your GitHub repository fork URL:

flux bootstrap git \
  --author-email=<YOUR-EMAIL> \
  --url=ssh://[email protected]/<YOUR-USERNAME>/gitops-istio \
  --branch=main \
  --path=clusters/my-cluster

The above command requires ssh-agent, if you're using Windows see flux bootstrap github documentation.

At bootstrap, Flux generates an SSH key and prints the public key. In order to sync your cluster state with git you need to copy the public key and create a deploy key with write access on your GitHub repository. On GitHub go to Settings > Deploy keys click on Add deploy key, check Allow write access, paste the Flux public key and click Add key.

When Flux has access to your repository it will do the following:

• installs Istio using the Istio base, istiod and gateway Helm charts
• waits for Istio control plane to be ready
• installs Flagger, Prometheus and Grafana
• creates the Istio public gateway
• creates the prod namespace
• creates the load tester deployment
• creates the frontend deployment and canary
• creates the backend deployment and canary

When bootstrapping a cluster with Istio, it is important to control the installation order. For the applications pods to be injected with Istio sidecar, the Istio control plane must be up and running before the apps.

With Flux v2 you can specify the execution order by defining dependencies between objects. For example, in clusters/my-cluster/apps.yaml we tell Flux that the apps reconciliation depends on the istio-system one:

apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 30m0s
  dependsOn:
    - name: istio-system
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps

Watch Flux installing Istio first, then the demo apps:

watch flux get kustomizations

You can tail the Flux reconciliation logs with:

flux logs --all-namespaces --follow --tail=10

List all the Kubernetes resources managed by Flux with:

flux tree kustomization flux-system

Istio customizations

You can customize the Istio installation using the Flux HelmReleases located at istio/system/istio.yaml:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: istio-gateway
  namespace: istio-system
spec:
  dependsOn:
    - name: istio-base
    - name: istiod
  # source: https://github.com/istio/istio/blob/master/manifests/charts/gateway/values.yaml
  values:
    autoscaling:
      enabled: true

After modifying the Helm release values, you can push the change to git and Flux will reconfigure the Istio control plane according to your changes.

You can monitor the Helm upgrades with:

flux -n istio-system get helmreleases --watch

To troubleshoot upgrade failures, you can inspect the Helm release with:

kubectl -n istio-system describe helmrelease istio-gateway

Flux issues Kubernetes events containing all the errors encountered during reconciliation. You could also configure Flux to publish the events to Slack, MS Team, Discord and others; please the notification guide for more details.

Istio control plane upgrades

Istio upgrades are automated using GitHub Actions and Flux.

When a new Istio version is available, the update-istio GitHub Action workflow will open a pull request with the manifest updates needed for upgrading Istio. The new Istio version is tested on Kubernetes Kind by the e2e workflow and when the PR is merged into the main branch, Flux will upgrade Istio on the production cluster.

Application bootstrap

When Flux syncs the Git repository with your cluster, it creates the frontend/backend deployment, HPA and a canary object. Flagger uses the canary definition to create a series of objects: Kubernetes deployments, ClusterIP services, Istio destination rules and virtual services. These objects expose the application on the mesh and drive the canary analysis and promotion.

# applied by Flux
deployment.apps/frontend
horizontalpodautoscaler.autoscaling/frontend
canary.flagger.app/frontend

# generated by Flagger
deployment.apps/frontend-primary
horizontalpodautoscaler.autoscaling/frontend-primary
service/frontend
service/frontend-canary
service/frontend-primary
destinationrule.networking.istio.io/frontend-canary
destinationrule.networking.istio.io/frontend-primary
virtualservice.networking.istio.io/frontend

Check if Flagger has successfully initialized the canaries:

kubectl -n prod get canaries

NAME       STATUS        WEIGHT
backend    Initialized   0
frontend   Initialized   0

When the frontend-primary deployment comes online, Flagger will route all traffic to the primary pods and scale to zero the frontend deployment.

Find the Istio ingress gateway address with:

kubectl -n istio-system get svc istio-ingressgateway -ojson | jq .status.loadBalancer.ingress

Open a browser and navigate to the ingress address, you'll see the frontend UI.

Canary releases

Flagger implements a control loop that gradually shifts traffic to the canary while measuring key performance indicators like HTTP requests success rate, requests average duration and pod health. Based on analysis of the KPIs a canary is promoted or aborted, and the analysis result is published to Slack.

A canary analysis is triggered by changes in any of the following objects:

• Deployment PodSpec (container image, command, ports, env, etc)
• ConfigMaps and Secrets mounted as volumes or mapped to environment variables

For workloads that are not receiving constant traffic Flagger can be configured with a webhook, that when called, will start a load test for the target workload. The canary configuration can be found at apps/backend/canary.yaml.

Pull the changes from GitHub:

git pull origin main

To trigger a canary deployment for the backend app, bump the container image:

yq e '.images[0].newTag="6.1.1"' -i ./apps/backend/kustomization.yaml

Commit and push changes:

git add -A && \
git commit -m "backend 6.1.1" && \
git push origin main

Tell Flux to pull the changes or wait one minute for Flux to detect the changes on its own:

flux reconcile source git flux-system

Watch Flux reconciling your cluster to the latest commit:

watch flux get kustomizations

After a couple of seconds, Flagger detects that the deployment revision changed and starts a new rollout:

$ kubectl -n prod describe canary backend

Events:

New revision detected! Scaling up backend.prod
Starting canary analysis for backend.prod
Pre-rollout check conformance-test passed
Advance backend.prod canary weight 5
...
Advance backend.prod canary weight 50
Copying backend.prod template spec to backend-primary.prod
Promotion completed! Scaling down backend.prod

During the analysis the canary’s progress can be monitored with Grafana. You can access the dashboard using port forwarding:

kubectl -n istio-system port-forward svc/flagger-grafana 3000:80

The Istio dashboard URL is http://localhost:3000/d/flagger-istio/istio-canary?refresh=10s&orgId=1&var-namespace=prod&var-primary=backend-primary&var-canary=backend

Note that if new changes are applied to the deployment during the canary analysis, Flagger will restart the analysis phase.

A/B testing

Besides weighted routing, Flagger can be configured to route traffic to the canary based on HTTP match conditions. In an A/B testing scenario, you'll be using HTTP headers or cookies to target a certain segment of your users. This is particularly useful for frontend applications that require session affinity.

You can enable A/B testing by specifying the HTTP match conditions and the number of iterations:

  analysis:
    # schedule interval (default 60s)
    interval: 10s
    # max number of failed metric checks before rollback
    threshold: 10
    # total number of iterations
    iterations: 12
    # canary match condition
    match:
      - headers:
          user-agent:
            regex: ".*Firefox.*"
      - headers:
          cookie:
            regex: "^(.*?;)?(type=insider)(;.*)?$"

The above configuration will run an analysis for two minutes targeting Firefox users and those that have an insider cookie. The frontend configuration can be found at apps/frontend/canary.yaml.

Trigger a deployment by updating the frontend container image:

yq e '.images[0].newTag="6.1.1"' -i ./apps/frontend/kustomization.yaml

git add -A && \
git commit -m "frontend 6.1.1" && \
git push origin main

flux reconcile source git flux-system

Flagger detects that the deployment revision changed and starts the A/B testing:

$ kubectl -n istio-system logs deploy/flagger -f | jq .msg

New revision detected! Scaling up frontend.prod
Waiting for frontend.prod rollout to finish: 0 of 1 updated replicas are available
Pre-rollout check conformance-test passed
Advance frontend.prod canary iteration 1/10
...
Advance frontend.prod canary iteration 10/10
Copying frontend.prod template spec to frontend-primary.prod
Waiting for frontend-primary.prod rollout to finish: 1 of 2 updated replicas are available
Promotion completed! Scaling down frontend.prod

You can monitor all canaries with:

$ watch kubectl get canaries --all-namespaces

NAMESPACE   NAME      STATUS        WEIGHT
prod        frontend  Progressing   100
prod        backend   Succeeded     0

Rollback based on Istio metrics

Flagger makes use of the metrics provided by Istio telemetry to validate the canary workload. The frontend app analysis defines two metric checks:

    metrics:
      - name: error-rate
        templateRef:
          name: error-rate
          namespace: istio-system
        thresholdRange:
          max: 1
        interval: 30s
      - name: latency
        templateRef:
          name: latency
          namespace: istio-system
        thresholdRange:
          max: 500
        interval: 30s

The Prometheus queries used for checking the error rate and latency are located at flagger-metrics.yaml.

Bump the frontend version to 6.1.2, then during the canary analysis you can generate HTTP 500 errors and high latency to test Flagger's rollback.

Generate HTTP 500 errors:

watch curl -b 'type=insider' http://<INGRESS-IP>/status/500

Generate latency:

watch curl -b 'type=insider' http://<INGRESS-IP>/delay/1

When the number of failed checks reaches the canary analysis threshold, the traffic is routed back to the primary, the canary is scaled to zero and the rollout is marked as failed.

$ kubectl -n istio-system logs deploy/flagger -f | jq .msg

New revision detected! Scaling up frontend.prod
Pre-rollout check conformance-test passed
Advance frontend.prod canary iteration 1/10
Halt frontend.prod advancement error-rate 31 > 1
Halt frontend.prod advancement latency 2000 > 500
...
Rolling back frontend.prod failed checks threshold reached 10
Canary failed! Scaling down frontend.prod

You can extend the analysis with custom metric checks targeting Prometheus, Datadog and Amazon CloudWatch.

For configuring alerting of the canary analysis for Slack, MS Teams, Discord or Rocket see the docs.

Getting Help

If you have any questions about progressive delivery:

• Invite yourself to the CNCF community slack and join the #flux and #flagger channels.
• Check out the Flux talks section and to see a list of online talks, hands-on training and meetups.

Your feedback is always welcome!

gitops-istio's Issues

Istio helm chart not installed

when running through your README, I can see the istio-system and the CRDs however istio isn't installed.

Update guide with Helm v3

Helm v3 removed the Tiller component. It would be nice to have the guide updated to use Helm v3 instead of the previous version based on Tiller.

Error when installing latest version

Hi,

I have tried to install latest version on GKE (1.13.6), and get the following error :

ts=2019-06-26T14:01:26.541674009Z caller=images.go:18 component=sync-loop msg="polling images"
ts=2019-06-26T14:01:26.633669856Z caller=images.go:24 component=sync-loop error="getting unlocked automated resources: duplicate definition of ':customresourcedefinition/sidecars.networking.istio.io' (in istio-system/crd-10.yaml and istio-system/crd-11.yaml)"

Could you please to take a look ? Thank you so much.

Best Regards,
VietNC

Istio telemetry keeps crashing after install

Hi,

I have tried to install latest version on GKE (1.13.6), component Flagger (0.16) and Istio (1.2.0) are up, but when I access dashboards, it cannot load information about namespace (but the namespace is existed on k8s) and some graphs is missing data as following attached images :

Hope you could help to take a look this issue. Thank you so much.

Best Regards,
VietNC

Unable to install after running the flux-init

I have a fresh install Kubernetes cluster on AWS EKS.

The problem occured when installing using flux-init.sh:

Installing Flux for ${REPO_URL}
Error: flux namespace does not exists

I guessed the issue is flux is not created

So, I need to execute kubectl create namespace flux before ./scripts/flux-init.sh [email protected]:<YOUR-USERNAME>/gitops-istio

Update helm repo address

Ref: fluxcd/helm-operator-get-started#28

Upgrade to Istio 1.3.0

I want to make a bugs learning model, but I need bugs data ,how can I find it

e2e tests fail after bootstrap

I think the reconciler needs to ignore my-cluster, but then I'm not sure where it gets the istio version?

small typo and a question

I think there is a typo in the readme, because the service for my minikube test is not istio-ingressgateway but

kubectl -n istio-system get svc istio-gateway -ojson | jq .status.loadBalancer.ingress

returning

[
  {
    "ip": "10.96.193.77"
  }
]

Aside from this irrelevant fix, I 've found very usefull your suggestion here and after

kubectl delete pod/istiod-7848cf56c8-t5fnw -n istio-system

all pods in istio-system started running. Good, however that comment is from 2019; is there any recent information on more automated management nowadays, without the need for manual pod deletion? If this is off-topic here due to being an issue related to Istio or something else, feel free to disregard.

My final question, how about changing the tag version in the frontend instead of the backend, so to make the effect more visible also from the UI?
Otherwise, the backend version appears to be non-discoverable from the user's perspective, doesn't it?

Resources requirements update on minikube

I was trying to deploy on minikube with 2 cpu and 4096MB memory and experienced this error on pod istio-telemetry:

Events:
  Type     Reason            Age                 From               Message
  ----     ------            ----                ----               -------
  Warning  FailedScheduling  11s (x23 over 21m)  default-scheduler  0/1 nodes are available: 1 Insufficient cpu.

Then, I tried to use minikube with 4 cpu instead and everything's fine.

There is a doc on Istio stating that it is recommended to use 4 cpu to start with minikube. Maybe an update on README is needed?

diff --git a/README.md b/README.md
index f661cd4..bd33e8f 100644
--- a/README.md
+++ b/README.md
@@ -23,7 +23,7 @@ Components:
 
 You'll need a Kubernetes cluster **v1.11** or newer with `LoadBalancer` support, 
 `MutatingAdmissionWebhook` and `ValidatingAdmissionWebhook` admission controllers enabled. 
-For testing purposes you can use Minikube with two CPUs and 4GB of memory. 
+For testing purposes you can use Minikube with four CPUs and 4GB of memory. 
 
 Install Flux CLI, Helm and Tiller:

Wrong version of the flagger-loadtester

I was trying to install your project and stuck on the failing pod flagger-loadtester.

When I checked the logs, it shows:
Failed to pull image "weaveworks/flagger-loadtester:0.21.1": rpc error: code = Unknown desc = Error response from daemon: manifest for weaveworks/flagger-loadtester:0.21.1 not found

I fixed it by changing the version through:
fluxctl release --workload=prod:deployment/flagger-loadtester \ --update-image=weaveworks/flagger-loadtester:0.12.1 --k8s-fwd-ns flux

Line issue:

gitops-istio/prod/loadtest/deployment.yaml

Line 21 in 2c4def1

image: weaveworks/flagger-loadtester:0.21.1

Update to Istio 1.5 and Flagger 1.0

Use Istio operator CRD instead of the deprecated Helm chart.

Unable to see any metrics on grafana dashboard

After following your instructions, I am not able to see any metrics on the 'Istio Canary' dashboard within grafana. I tried opening prometheus and running a query for istio_requests_total just to see the raw time series data, but there is nothing there. It appears that istio is not collecting any metrics? Or possibly that the load test is not actually sending any requests through the system?

Do I need to run any commands to initiate the load test?