How can I upgrade tiller on ACS?

[azure]

How can I upgrade tiller on ACS?

[berndverst]

Documenting my single command workaround based on comments above (make sure to replace the host with your ACS master node)

ssh [email protected] sudo sed -i s/'2.5.1'/'2.6.0'/g /etc/kubernetes/addons/kube-tiller-deployment.yaml && helm init --upgrade

[lrills]

Any update on this? Cluster in Southeast Asia still can't upgrade with helm init --upgade now.

[bacongobbler]

@berndverst I tried your example and it worked, however I would suggest restarting the kube-addon-manager pod by deleting it from the kube-system namespace, e.g.

$ kubectl -n kube-system delete po kube-addon-manager-k8s-master-a637ed6-0

This will force a new addon-manager pod to boot and reconcile all the resources managed by the addon manager... Including tiller. This will force a fresh tiller v2.6.0 instance to be deployed.

[JackQuincy]

This appears to be because of how we are defining our tiller addon. We have it marked as a cluster service. Which means Kuberentes doesn't want to allow updates to it through the kubernetes api. Instead it wants the updates to happen on the file in the master. We could change to using addonmanager.kubernetes.io/mode=EnsureExists . @anhowe @seanknox @jackfrancis thoughts?

relevant Kubernetes documentation
https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/addon-manager/README.md

[JackQuincy]

It will be in the v2 regions in a week or two if it isn't already. In the v1 regions it will be next month.

[JackQuincy]

Definetly in ACSv2. V1 is still a ways out we have had a deployment having other issues that is delaying this some.

[bacongobbler]

I’m going to reopen this for visibility. I’m still making weekly references to this ticket in #helm-users. :)

[seanknox]

addonmanager.kubernetes.io/mode=EnsureExists appears to be right call here. Agree @lachie83?

[JackQuincy]

Also what do we think about doing this for dashboard, heapster, and the storage classes addons? Do we want them static or do we want customers to be able to modify them through the api server

[seanknox]

Good call for dashboard and heapster, though my hunch without thinking through much is that we should err on the more prescriptive end with storage classes.

[JackQuincy]

I think I agree on not letting people shoot themselves in the foot with storage classes. Mostly I was wanting to look at all the extensions and decide if we want to update them while doing this.

[JackQuincy]

@bacongobbler if it wasn't clear there is a work around by updating the add-on file for tiller, /etc/kubernetes/addons/kube-tiller-deployment.yaml, to be the newer spec for tiller on each master.

[lachie83]

Confirming everything @JackQuincy @seanknox have said.

[bacongobbler]

I was able to work around this by deploying a hacked up acs-engine cluster with Tiller v2.6.0 but I'll keep that in mind for next time. Thanks @JackQuincy @seanknox @lachie83 for your help!

[seanknox]

JFYI we have the long-term fix for this slated to go into acs-engine soon: Azure/acs-engine#1319

[stuartleeks]

What is the timeline for this being fixed in ACS?

[subho007]

Any updates on this?

[berndverst]

@lrills I manually had to update the add on definition on the Kubernetes master again as a workaround. Like so:

ssh [email protected] sudo sed -i s/'2.6.0'/'2.7.0'/g /etc/kubernetes/addons/kube-tiller-deployment.yaml && helm init --upgrade

No other method has reliably worked for me.