azure/kubernetes-hackfest

Delivering modern cloud-native applications with open source technologies on Azure Kubernetes Service

Prerequisites

The following are the basic requirements to start the labs. Individual labs may have other requirements that will be listed within the lab.

• Azure Account Azure Portal
• Git Git SCM
• Azure Cloud Shell https://shell.azure.com

Application

Labs

Start with:

1. Create AKS Cluster
2. Build Application Components and Prerequisites
3. Helm Setup and Deploy Application

Additional exercises (in no order):

• CI/CD Automation
• Networking
• Security
• Monitoring and Logging
• Scaling Cluster and Applications
• Azure Container Instances and AKS Virtual Nodes
• Azure Dev Spaces
• Service Mesh
• Best Practices for Cluster Operators
• Best Practices for App Developers
• Calico lab exercise
• Kubecost - Kubernetes Cost Insights

Coming soon:

• Storage
• Service Broker
• Using OpenFaaS on AKS

Contributing

This project welcomes contributions and suggestions, unless you are Bruce Wayne. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

kubernetes-hackfest's Issues

Consider porting to Azure Maps

Since this is an Azure sample, would be great if it used Azure Maps. Azure Maps has most of the same features and performance benefits of Mapbox GL JS.

Need to add resource requests and limits to the Helm charts

Azure Pipelines helm upgrade fails due to unscoped values in commandline overrides

In Lab: Azure DevOps CI/CD section Create Deployment Pipeline, step 14 the Set Values line example is misleading as it is not scoped properly (missing "deploy." prefix) which leads to a deploy failure:

Upgrade "service-tracker-ui" failed: timed out waiting the condition

Change the line:

Eg - acrServer=acrhackfestbrian123.azurecr.io,imageTag=vsts-$(Build.BuildId)

to

Eg - acrServer=deploy.acrhackfestbrian123.azurecr.io,deploy.imageTag=vsts-$(Build.BuildId)

Update `cd links` to include entire path

Need to update links so they just include entire path. This will avoid users needing to cd throughout the labs

Automate changing charts values.yaml

Need to make this easer as it's error prone for a user to update the values.yaml. I think sed would be able to change all values.yaml in a one-liner

Jenkins lab need instructions for sp

We should have the users create a new service principal for use with Jenkins. It would be helpful to provide CLI steps for the create and where to find the 4 values.

Lab: Helm Setup And Deploy

Step 5 doesn't contain instructions on obtaining CosmosDB variable inputs.

Add Jenkins instructions to ensure update helm yaml files

in the Jenkins Lab (https://github.com/Azure/kubernetes-hackfest/blob/master/labs/cicd-automation/jenkins/README.md)

Lab does not mention to update acrServer in yaml files.

Ensure CosmosDB URI example

Create AKS cluster with monitoring addon in step 8 requires workspace already defined

In Lab 1 - Create AKS cluster, the instructions mention AKS cluster to be created with monitoring addon. But log analytics workspace has been created yet and we need to pass workspace id in the argument, which is missing. Hence AKS creation fails.

CICD Lab - Jenkins: Jenkins needs update after first load for pipelines to work

After following the directions I noticed I was getting the following error on build:
Error: [Pipeline] End of Pipeline
java.lang.NoSuchMethodError: No such DSL method 'pipeline' found among steps [archive, bat,.....
Needed to fix the pipeline plugin

After a little digging I realized that the Jenkins environment had an error an was pending some updates. Once I ran the updates recommended by the jenkins portal I was good to go.

Verify selected subscription is the correct one (if you have multiple subscriptions)

In Lab 1: Create AKS Cluster, Step 4, it states:

Note: In the cloud shell, you are automatically logged into your Azure subscription.

However, this does not mean the subscription you want to use will be selected if you have multiple subscription.

I have two subscriptions: "Visual Studio Enterprise" and "Microsoft Azure Internal Consumption". The first one was selected for me but I wanted to use the second subscription.

I suggest adding a verification step to list subscriptions:

# View subscriptions
az account list

# Verify selected subscription
az account show

# Set correct subscription (if needed)
az account set --subscription <subscription_id>

# Verify correct subscription is now set
az account show

Update Security AD Lab commands

You declare the AZUREAD-TENANT-ID twice, and there's no "export" in front of the commands

Issue retrieving cosmosdb connect string

this command az cosmosdb list-connection-strings --name $COSMOSNAME --resource-group $RGNAME returns nothing. nada. blank. null. empty. like completely zero.

Lab 1 / kubectl cluster-info dump : error: missing apiVersion or kind and cannot assign it; no kind is registered for the type core.NodeList

$ kubectl cluster-info dump
error: missing apiVersion or kind and cannot assign it; no kind is registered for the type core.NodeList
$ kubectl version --short
Client Version: v1.11.0
Server Version: v1.10.3

Recommendation : Update docs and deploy Kubernetes version 1.11.2 or 1.11.0.

Upgrade to Kubernetes version to be deployed to 1.11.5

Please update Lab 1, so that Kubernetes 1.11.5 is deployed else people will experience the following error.

$ az aks create -n $CLUSTERNAME -g $RGNAME -k 1.11.3 --service-principal $APPID --client-secret $CLIENTSECRET --generate-ssh-keys -l southeastasia --node-count 3 --enable-addons http_application_routing,monitoring
Operation failed with status: 'Bad Request'. Details: The value of parameter orchestratorProfile.OrchestratorVersion is invalid.

Lab: Helm Setup and Deploy Application - Helm version is wrong in docs

Helm version in guide specifies 2.9.1 should be 2.10.0

Lab 3 / Step 4 - updating values.yaml to point to newly created ACR

Would'nt it be simpler to say, change the ACR configuration within the following files to point to your newly created ACR and update the version number.

It took me a few minutes of looking for these files and re-reading what was being asked, let's make it dead simple.

./service-tracker-ui/values.yaml
./data-api/values.yaml
./flights-api/values.yaml
./weather-api/values.yaml
./quakes-api/values.yaml

Lab: Build Application Components - ACR Registration guide

The instructions in Step 2

Run bash script to authenticate with Azure Container Registry from AKS
Running this script will grant the Service Principal created at cluster creation time access to ACR.
sh reg-acr.sh

Should also include instructions to

• change directory into the lab directory

• instructions to edit the script to put in values for RG, AKS name and ACR name, or the script should use the predefined values

Refactor CosmosDB Instructions

Add setup for CosmosDB database

Lab 2 will have issues if cloud shell times out

They went to create ACR and the variable $UNIQUE_SUFFIX became empty. Thus the az acr create failed. Users will need to reset this variable to match the step in lab 1.

Probably a better way to do this.

Lab 4 / URL Update for VSTS

The link to creating a VSTS account does'nt work due to the re-branding of VSTS to Azure DevOps so I presume that you want to update it to this one.

https://docs.microsoft.com/en-us/azure/devops/organizations/accounts/create-organization-msa-or-work-student?view=vsts

Old link references in the docs :
https://docs.microsoft.com/en-us/vsts/organizations/accounts/create-account-msa-or-work-student?view=vsts

Add these instructions to the Jenkins Lab

These would be for finding the Client and Tenant IDs for entering into Jenkins:

Client (or Application ID) - In the Azure console, search for "Azure Active Directory" and click on it. In the left pane, under Manage, select "App registrations", then click "View all applications" in the right tab. Once done, select your Display Name and double-click. The Application ID will be listed in the pane.

Tenant ID (or Directory ID) - In the Azure CLI, run the following command:

az account show

The Tenant ID will be shown in the field "tenantID".

Lab 3 / Step 5 - CosmosDB URL and AppInsights config + K8s secrets

Would'nt it be easier for the user to run a couple of commands and extract the CosmosDB config.

MONGODB_URI

az cosmosdb list-connection-strings --name $COSMOSNAME --resource-group $RGNAME

MONGODB_USER

az cosmosdb show --name $COSMOSNAME --resource-group $RGNAME --query "name" -o tsv

MONGODB_PASSWORD

az cosmosdb list-keys --name $COSMOSNAME --resource-group $RGNAME --query "primaryMasterKey" -o tsv

APPINSIGHTS_INSTRUMENTATIONKEY

az resource list --namespace microsoft.insights --resource-type components --query [*].[id] --out tsv
az resource show --id "/subscriptions/1234fd22-ffff-4444-8888-ab5083888888/resourceGroups/kubernetes-hack
fest/providers/microsoft.insights/components/darren8176" --query properties.InstrumentationKey --o tsv

Once the secret has been stored in Kubernetes, please show how easy it is to read this data and howto decode it. This is particularly useful for troubleshooting.

kubectl get secret cosmos-db-secret -o yaml
kubectl get secrets cosmos-db-secret -o jsonpath --template '{.data.user}' | base64 -d

helm rbac config's are redundant

we need to make this more clear in the labs

Lab 1 / Adding variable to .bashrc does'nt point to users home directory

Lab 1

echo export UNIQUE_SUFFIX=$UNIQUE_SUFFIX >> .bashrc

Should be the following, to write into .bashrc in the users home directory, not in current directory which is kubernetes-hackfest/labs/create-aks-cluster.

echo export UNIQUE_SUFFIX=$UNIQUE_SUFFIX >> ~/.bashrc

Some browsers may not load maps without overriding GPU blacklist

May be good to add a note that the maps may not load on some browsers by default (ex. Chromium). This is because GPU drivers get blacklisted by default in some browsers. In Chromium you can go to chrome://flags and enable "Override software rendering list".

Issue with AKS creation

we are auto generating the service principal and it does not have rights to their ACR. This ends up to image pull failures. we should have them create a SP manually, assign it rights to the RG, and use it in the AKS create

there is also a bigger problem and the cred failure being cached. need to restart nodes or something to get them to work.

In Lab 1, there is a "Lab 2" towards the end after the Docs

The namespaces stuff is called "Lab 2" and it should probably be lab 1a. Our users got confused.

Broken link in the networking lab

oh snap, 502 Bad Gateway

I did the Configure Ingress Controller lab before Configure Network Policy.

At the "Test out the Application by going to the Dashboard..." step of the Configure Network Policy lab, I immediately get a 502 Bad Gateway error, as my service-tracker-ui service is now of type ClusterIP, and I access the ui via https://[myacr].eastus.cloudapp.azure.com/ui#/dashboard, not from a LoadBalancer IP address.

Perhaps the README.md should note this distinction?

Security lab needs _ in variables

https://github.com/Azure/kubernetes-hackfest/blame/master/labs/security/create-rbacwithazuread-cluster/README.md#L36

Lab 1 - Clean up nginx-limittest and nginx-quotatest pods

At the end of Lab 1: Create AKS Cluster, the two pods for testing limits and quotas are not cleaned up.

Suggestion: add a final step to delete the pods

6. Clean up pods

kubectl delete po nginx-limittest nginx-quotatest -n dev

Break up bash commands into single line

Break up bash commands into single line. Putting bash commands with output can confuse users on what command they should run next.

Lab 4 [VSTS] / Step 9 - acr build in Azure DevOps

Original commands in docs is this :

export ACRNAME=<replace>
export IMAGETAG=vsts-$(Build.BuildId)

az acr build -t hackfest/node-data-api:v1 -r $ACRNAME --no-logs ./app/node-data-api
az acr build -t hackfest/node-flights-api:v1 -r $ACRNAME --no-logs ./app/node-flights-api
az acr build -t hackfest/web-ui:v1 -r $ACRNAME --no-logs ./app/web-ui 

Should'nt we be building all code (or just the UI) and then configuring the version number of each container image.

export ACRNAME=<replace>
export IMAGETAG=vsts-$(Build.BuildNumber)
az acr build -t hackfest/data-api:$IMAGETAG -r $ACRNAME --no-logs ./app/data-api
az acr build -t hackfest/flights-api:$IMAGETAG -r $ACRNAME --no-logs ./app/flights-api
az acr build -t hackfest/quakes-api:$IMAGETAG -r $ACRNAME --no-logs ./app/quakes-api
az acr build -t hackfest/weather-api:$IMAGETAG -r $ACRNAME --no-logs ./app/weather-api
az acr build -t hackfest/service-tracker-ui:$IMAGETAG -r $ACRNAME --no-logs ./app/service-tracker-ui

I had issues with the helm deployment, using the --force option (tickbox in Releases) did not resolve this.

2018-09-16T06:16:33.8883231Z [command]/usr/local/bin/helm upgrade --install --wait Release-3 /home/vsts/work/r1/a/_vsts-aks-CI/charts/service-tracker-ui
2018-09-16T06:16:39.4116659Z Error: release Release-3 failed: services "service-tracker-ui" already exists
2018-09-16T06:16:39.4133827Z Release "Release-3" does not exist. Installing it now.
2018-09-16T06:16:39.4252737Z ##[error]Error: release Release-3 failed: services "service-tracker-ui" already exists

Change API services to ClusterIP in helm charts

in lab 1, kubectl cluster-info dump is confusing

We should reword this a bit ... not sure what to change it to, but it is confusing in its current context

Modify labs to use variables stored in .bashrc

we need to tweak instructions where variables such as $RGNAME are used.

Path should be just "/" without UI so that service loads properly.

kubernetes-hackfest/labs/networking/ingress/service-tracker-ingress.yaml

Line 17 in d4839bd

- path: /ui

Add in Account Show and Set

This is to ensure a user is using the right subscription.

Consider using separate, unique RG for each user

Consider changing to:

# Set Resource Group Name
RGNAME=$UNIQUE_SUFFIX

Lab 3 / Step 7 - curl hitting refresh endpoint will not output any data

When connecting to the endpoint using curl on /refresh, you don't see any json output which I understand is expected behaviour however I found this confusing and wondered if something were broken.... Would be awesome if you can re-write this section to make this clearer.

Also, the target TCP ports on the Azure LoadBalancer are different for flights-api quakes-api weather-api.
Can you please highlight this, as i'm sure that this will cause a lot of confusion.

curl http://<EXTERNAL-IP>:3003/refresh
<no output>
curl http://<EXTERNAL-IP>:3012/refresh
<no output>
curl http://<EXTERNAL-IP>:3015/refresh
<no output>

When connecting to the endpoint IP directly using curl, you do see output.

curl http://<EXTERNAL-IP>:3015
"message":"api default endpoint for weather api","payload":{}}

Jenkins issue

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://kubernetes.default/api/v1/namespaces/default/pods. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "aks-hackfest-5xt6m-9d308" is forbidden: error looking up service account default/jenkins-jenkins: serviceaccount "jenkins-jenkins" not found.
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:472)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:409)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:381)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:344)
at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:227)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:766)
at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:335)
at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:105)
at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:292)
at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Authenication key in code

Looks like your Mapbox key is in the code. You might want to remove it some no one steals it and runs up charges on it.

In lab 1 (part 2), the create namespaces command does not provide a path to the yaml

when they run kubectl apply -f create-namespaces.yaml they are not in the right directory. Need to point them to the lab folder.

Lab 2 / reg-acr.sh

Lab 2

1.Please add in "cd" below at the beginning of Lab 2. I know this is implied however people will still ask this question.

"cd kubernetes-hackfest/labs/build-application"

2.I'd suggest changing the variables in reg_acr.sh to the following or updating this script

AKS_RESOURCE_GROUP=$RGNAME
AKS_CLUSTER_NAME=$CLUSTERNAME
ACR_RESOURCE_GROUP=$RGNAME
ACR_NAME=$ACRNAME

Issue with Jenkins lab and external IP

On line 82 of README, the command has jenkins-jenkins but it looks like it should just be jenkins as below:

printf $(kubectl get secret --namespace default jenkins-jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo

Links to Next Lab

Links to Next Lab

Lab 3 / Step 4 - updating values.yaml to point to newly created ACR

Would'nt it be simpler to say, change the ACR configuration within the following files to point to your newly created ACR.
It took me a few minutes of looking for these files and re-reading what was being asked, let's make it dead simple.

./service-tracker-ui/values.yaml
./data-api/values.yaml
./flights-api/values.yaml
./weather-api/values.yaml
./quakes-api/values.yaml

helm-setup application insights - chart location

need to describe / or link to / the chart locations - and explain how this works