[kubernetes] k8s cluster starts working after nodes are restarted

[azure]

[kubernetes] k8s cluster starts working after nodes are restarted

[colemickens]

@olostan Your SP is very likely misconfigured. Please check the troubleshooting steps here: https://github.com/Azure/acs-engine/blob/master/docs/kubernetes.md#troubleshooting

[colemickens]

How much time elapsed between the SP creation and the reboot?

Also, I know it's a weird question, but what timezone did you execute the SP creation command in?

[kim0]

@colemickens .. Let me answer the second question first. I executed sp creation commands, on my laptop in UTC+2 TZ!

For the first question, it's not entirely clear to me. I created the SP using the new python az cli. Then, I submitted the k8s deployment through the azure portal. I guess it took like 5 mins to deploy. After that, I ssh'ed into master. It and all agents were not working kubectl get nodes not returning. Restart api server docker container, made it responsive. Rebooting agent-1 made it ready! I still did not reboot agents-{2&3}, and they are still in the "Not-ready" state, if you want to ssh to them, I'm Ok. Otherwise, if I'm free to reboot them, let me know as well. Thanks!

[kim0]

Rebooted remaining nodes, and they came up in ready state

[olostan]

@kim0 How you get api server restarted? I've ssh-ed to master, tried to do same as you, but:

root@k8s-master-F02F8C45-0:~# docker ps
CONTAINER ID        IMAGE                                             COMMAND                  CREATED             STATUS              PORTS               NAMES
9fb65bdb36c6        gcr.io/google_containers/hyperkube-amd64:v1.4.6   "/hyperkube kubelet -"   19 minutes ago      Up 19 minutes                           angry_thompson
root@k8s-master-F02F8C45-0:~# kubectl get pods
^C
root@k8s-master-F02F8C45-0:~# docker restart 9fb65bdb36c6
9fb65bdb36c6
root@k8s-master-F02F8C45-0:~# docker ps
CONTAINER ID        IMAGE                                             COMMAND                  CREATED             STATUS              PORTS               NAMES
9fb65bdb36c6        gcr.io/google_containers/hyperkube-amd64:v1.4.6   "/hyperkube kubelet -"   20 minutes ago      Up 5 seconds                            angry_thompson
root@k8s-master-F02F8C45-0:~# kubectl get pods
Unable to connect to the server: dial tcp 13.95.157.85:443: i/o timeout

[olostan]

@colemickens wow... thnx. That helps - seems I really have

Unable to construct api.Node object for kubelet:
 failed to get external ID from cloud provider: compute.VirtualMachinesClient#Get:
 Failure responding to request: StatusCode=403
 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" 
Message="The client '<guid>' with object id '<guid>' does not have authorization to
 perform action 'Microsoft.Compute/virtualMachines/read' over scope '/subscriptions/<guid>/resourceGroups/edicircle/providers/Microsoft.Compute/virtualMachines/k8s-master-f02f8c45-0'."

Sorry if it is not correct place (however could be useful for others if they have same problem, but is there any clue how to add those permissions?

Actually I have created cluster exactly one-to-one as on this video: https://www.youtube.com/watch?v=nhY9XdzNbbY with az acs create ... that should should create SP as I understand....

[colemickens]

@olostan If you used az acs create, you are correct, you shouldn't be experiencing this issue.

Can you please detail the exact command you ran (possibly looking through your shell history) and also paste the full output from az --version? Thanks!

(We might end up moving this over to https://github.com/Azure/azure-cli ...)

[olostan]

Created Azure/azure-cli#1620
@colemickens if you have any ideas how I can manually add authorisation rights please share! thnx

[anhowe]

auth rights to resource groups should be able to be added through the Azure Portal.

[anhowe]

Please re-open if you encounter this issue again, since the latest az should fix this.