Kubernetes vm scale up problem

[azure]

Kubernetes vm scale up problem

[canthefason]

I am still investigating this issue. To be more specific, after this scale up request, I can still reach the apiserver via the insecure port. It looks like a problem in network level. I will keep updating with my findings.

[canthefason]

yes, it is definitely related to the VM types. When I try to create my cluster from the UI, it sets the agent VM value as Standard_DS1. It works fine for initial creation, but it does not work at all when it comes to scaling up the agents.

Then I tried to create my cluster with azure-cli. Initially, I set Standard_DS1 as the VM value, and it did not get validated. Later on, I picked the same VM value with example/kubernetes.json as Standard_D2_v2, and both cluster creation and scaling worked like a charm.

[colemickens]

Are you using ACS-Engine, or are you using the actual Container Service and trying to scale it via a PUT?

[canthefason]

In the beginning, I used UI. But now I am using ACS-Engine.

There is one last thing I realized that might be helpful for you: I followed these steps to generate necessary templates. After that, I checked azuredeploy.json, and I even did not see Standard_DS1 under "agentpool1VMSize.allowedValues".

[colemickens]

cc: @anhowe for the question about the list of VM sizes.

Scaling in ACS itself is not yet supported, and in fact I believe we will soon be blocking updates until we implement scale up/scale down for Kubernetes.

With ACS-Engine, you can use the outputed "apimodel.json" file, change the desired values, and then re-deploy the template. There is an issue (#108) to document this process which we will do soon.

Any estimation by when we can expect scaling functionality on Azure ?
In the meantime, is there any documentation on how to add nodes manually ?
Do you know if this will need to redeploy the cluster or if the already created cluster can be "upgraded" to support new functionalities ?

[colemickens]

@kaubry In the reply just above yours I described how acs-engine can be used to scale up/down.

Do you know if this will need to redeploy the cluster or if the already created cluster can be "upgraded" to support new functionalities ?

You don't have to redeploy, but upgrading Kubernetes versions is currently a manual process. You can read a bit about that here. colemickens/azure-kubernetes-status#15 We should document this in this acs-engine repo soon.

CC: @rgardler for timelines on scaling / upgrades. Not sure what sort of timeline specifics we're offering right now.

@colemickens Thanks you for you quick reply. I manage to make everything work (upscale and upgrading Kubernetes version) but for scaling down, acs-engine doesn't seems to handle it. I need to manually remove the created VM and the corresponding network interface. Is it the right way to proceed or am I missing something ?

I tried as well to change the size of 1 node only (if I want some nodes to have more power) and when I run the script to scale up my cluster. It changes my up-sized node back to default. Anything I can do to avoid this behaviour ?

[colemickens]

@kaubry You are correct.

[blocksninja]

but for scaling down, acs-engine doesn't seems to handle it. I need to manually remove the created VM and the corresponding network interface. Is it the right way to proceed or am I missing something ?

Also, don't forget to download the OS disk of the agent. Otherwise you'll get a "TargetDiskBlobAlreadyExists" error when scaling up again.

[colemickens]

Thanks @timbrand , we need to document this process and that's a great point to include.

[anhowe]

To avoid the "TargetDiskBlobAlreadyExists", you will need to choose the option to delete the blob, when scaling down.

[anhowe]

@JackQuincy - this will be fixed when you submit https://github.com/Azure/acs-engine/pull/366/files.

[rocketraman]

The documentation added from pull #366 states that the NSG has to be removed, and the diff of kubernetes_template.json and kubernetes_update_template.json @ https://github.com/Azure/acs-engine/tree/master/examples/scale-up does show this removal.

However, I just tried a scale-up, and it worked with just changing the count and adding the offset parameters -- there was no need to remove the NSG.

[JackQuincy]

I should of been more specific if you don't remove the NSG any public services on the cluster will go unavailable for a period of between 2-10 mins is what I've seen reported. Get Outlook for iOS<https://aka.ms/o0ukef>

…

________________________________ From: Raman Gupta <[email protected]> Sent: Tuesday, April 4, 2017 5:49:23 PM To: Azure/acs-engine Cc: Jack Quincy; Mention Subject: Re: [Azure/acs-engine] Kubernetes vm scale up problem (#109) The documentation added from pull #366<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Facs-engine%2Fpull%2F366&data=02%7C01%7Cjaquincy%40microsoft.com%7C855e7da8faec4649c37d08d47bbd9aae%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636269501653841638&sdata=%2BApV7SbR4w9T7O%2Badzq6EuGgRaJa7FEy%2FnNmN3CLJq4%3D&reserved=0> states that the NSG has to be removed, and the diff of kubernetes_template.json and kubernetes_update_template.json @ https://github.com/Azure/acs-engine/tree/master/examples/scale-up<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Facs-engine%2Ftree%2Fmaster%2Fexamples%2Fscale-up&data=02%7C01%7Cjaquincy%40microsoft.com%7C855e7da8faec4649c37d08d47bbd9aae%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636269501653841638&sdata=kAEZ%2FBmPTwqdzb%2Bbjpes0Wu46gftJRcjDdr5popKpvg%3D&reserved=0> does show this removal. However, I just tried a scale-up, and it worked with just changing the count and adding the offset parameters -- there was no need to remove the NSG. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FAzure%2Facs-engine%2Fissues%2F109%23issuecomment-291693290&data=02%7C01%7Cjaquincy%40microsoft.com%7C855e7da8faec4649c37d08d47bbd9aae%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636269501653841638&sdata=aHOQ%2BAkkj2rBIUnoye%2B%2Bo9S8D4WbedbvnOEMmklTeTU%3D&reserved=0>, or mute the thread<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAUBiNPbmR7X7KQc7kmTwgHdtRyPvd_xdks5rsuUTgaJpZM4K2MJF&data=02%7C01%7Cjaquincy%40microsoft.com%7C855e7da8faec4649c37d08d47bbd9aae%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636269501653841638&sdata=iGyXboD54nADFEnPLJkUvAGySiZcXy07Mkv5XiXY8WU%3D&reserved=0>.

[rocketraman]

if you don't remove the NSG any public services on the cluster will go unavailable for a period of between 2-10 mins is what I've seen reported.

Thanks for the clarification.

Would love to see a more permanent fix for the scaling up/down problem. This is certainly a key aspect of running a container orchestrator like Kubernetes that should be seamless.

[anhowe]

Closing since main issue is solved.