Kubeip should update the node it is running on with care

[doitintl]

Kubeip should update the node it is running on with care

[avivl]

Thanks again.
I think anti-affinity is a good approch
In the coming month, I'll not have time to work on it (a lot of traveling... and other work commitments).
If you'll create a PR I'll find a time to review it.

[pdecat]

Here are some more test results with more detailed logs and nanosecond resolution (code at https://github.com/pdecat/kubeip/tree/8234ba226619d3b917d356a76741b5aa0a6ae0de).

With Private Google Access disabled:

time="2018-10-03T16:55:11.272172478Z" level=info msg="kubeIP is starting" Build Date="2018-10-03-15:44" Cluster name=test-pdecat-kubeip Project name=myproject Version=8234ba226619d3b917d356a76741b5aa0a6ae0de
time="2018-10-03T16:55:11.277366376Z" level=info msg="Starting forceAssignment" function=forceAssignment pkg=kubeip
time="2018-10-03T16:55:11.278034426Z" level=info msg="Starting kubeip controller" pkg=kubeip-node
time="2018-10-03T16:55:11.279096626Z" level=info msg="Collecting Node List..." function=processAllNodes pkg=kubeip
time="2018-10-03T16:55:11.37830437Z" level=info msg="kubeip controller synced and ready" pkg=kubeip-node
time="2018-10-03T16:55:11.716145551Z" level=info msg="Found un assigned node gke-test-pdecat-kubei-cluster1-pool-a-073827fa-tjc9" function=processAllNodes pkg=kubeip
time="2018-10-03T16:55:11.716440622Z" level=info msg="Working on gke-test-pdecat-kubei-cluster1-pool-a-073827fa-tjc9 in zone europe-west1-b" function=Kubeip pkg=kubeip
time="2018-10-03T16:55:11.721135184Z" level=info msg="Found node without tag gke-test-pdecat-kubei-cluster1-pool-a-073827fa-tjc9" function=assignMissingTags pkg=kubeip
time="2018-10-03T16:55:12.049512096Z" level=info msg="Found reserved address 35.233.39.239" function=replaceIP pkg=kubeip
time="2018-10-03T16:55:13.129994201Z" level=info msg="Access config deletion in progress for instance gke-test-pdecat-kubei-cluster1-pool-a-073827fa-tjc9" function=replaceIP pkg=kubeip
time="2018-10-03T16:55:20.21601361Z" level=info msg="Access config deleted for instance gke-test-pdecat-kubei-cluster1-pool-a-073827fa-tjc9" function=replaceIP pkg=kubeip
time="2018-10-03T16:55:20.858648237Z" level=info msg="Access config creation in progress for instance gke-test-pdecat-kubei-cluster1-pool-a-073827fa-tjc9 with IP 35.233.39.239" function=replaceIP pkg=kubeip
time="2018-10-03T16:55:25.674663578Z" level=info msg="Replaced IP for gke-test-pdecat-kubei-cluster1-pool-a-073827fa-tjc9 zone europe-west1-b new ip 35.233.39.239" function=replaceIP pkg=kubeip
time="2018-10-03T16:55:38.884293791Z" level=error msg="Get https://10.39.240.1:443/api/v1/nodes?labelSelector=kubip_assigned%3D35-233-39-239: read tcp 10.36.1.8:60112->10.39.240.1:443: read: connection reset by peer"
time="2018-10-03T16:55:38.885531902Z" level=info msg="Tagging node gke-test-pdecat-kubei-cluster1-pool-a-073827fa-tjc9 as 35.233.39.239" function=tagNode pkg=kubeip

Notice that:

• the Replaced IP for message is here
• the one failing call is one to the kubernetes API to retrieve nodes in GetNodeByIP (not sure why this one failed)
• the tagging occurred

With Private Google Access enabled:

time="2018-10-03T16:34:53.609959393Z" level=info msg="kubeIP is starting" Build Date="2018-10-03-15:44" Cluster name=test-pdecat-kubeip Project name=myproject Version=8234ba226619d3b917d356a76741b5aa0a6ae0de
time="2018-10-03T16:34:53.613011245Z" level=info msg="Starting forceAssignment" function=forceAssignment pkg=kubeip
time="2018-10-03T16:34:53.613162179Z" level=info msg="Starting kubeip controller" pkg=kubeip-node
time="2018-10-03T16:34:53.614397466Z" level=info msg="Collecting Node List..." function=processAllNodes pkg=kubeip
time="2018-10-03T16:34:53.713606984Z" level=info msg="kubeip controller synced and ready" pkg=kubeip-node
time="2018-10-03T16:34:54.015584784Z" level=info msg="Found un assigned node gke-test-pdecat-kubei-cluster1-pool-a-073827fa-kz0b" function=processAllNodes pkg=kubeip
time="2018-10-03T16:34:54.015811129Z" level=info msg="Working on gke-test-pdecat-kubei-cluster1-pool-a-073827fa-kz0b in zone europe-west1-b" function=Kubeip pkg=kubeip
time="2018-10-03T16:34:54.02065915Z" level=info msg="Found node without tag gke-test-pdecat-kubei-cluster1-pool-a-073827fa-kz0b" function=assignMissingTags pkg=kubeip
time="2018-10-03T16:34:54.328296892Z" level=info msg="Found reserved address 35.233.39.239" function=replaceIP pkg=kubeip
time="2018-10-03T16:34:55.509424755Z" level=info msg="Access config deletion in progress for instance gke-test-pdecat-kubei-cluster1-pool-a-073827fa-kz0b" function=replaceIP pkg=kubeip
time="2018-10-03T16:35:02.631003629Z" level=info msg="Access config deleted for instance gke-test-pdecat-kubei-cluster1-pool-a-073827fa-kz0b" function=replaceIP pkg=kubeip
time="2018-10-03T16:35:03.274858087Z" level=info msg="Access config creation in progress for instance gke-test-pdecat-kubei-cluster1-pool-a-073827fa-kz0b with IP 35.233.39.239" function=replaceIP pkg=kubeip
time="2018-10-03T16:35:05.632233883Z" level=error msg="ZoneOperations.Get \"Get https://www.googleapis.com/compute/v1/projects/myproject/zones/europe-west1-b/operations/operation-1538584502639-577559ebc9998-03ffb820-78e87e27?alt=json: read tcp 10.36.0.7:51334->108.177.15.95:443: read: connection reset by peer\" operation-1538584502639-577559ebc9998-03ffb820-78e87e27"
time="2018-10-03T16:35:05.632307834Z" level=info msg="Replaced IP for gke-test-pdecat-kubei-cluster1-pool-a-073827fa-kz0b zone europe-west1-b new ip 35.233.39.239" function=replaceIP pkg=kubeip
time="2018-10-03T16:35:18.840329311Z" level=error msg="Get https://10.39.240.1:443/api/v1/nodes?labelSelector=kubip_assigned%3D35-233-39-239: read tcp 10.36.0.7:44136->10.39.240.1:443: read: connection reset by peer"
time="2018-10-03T16:35:18.84173178Z" level=info msg="Tagging node gke-test-pdecat-kubei-cluster1-pool-a-073827fa-kz0b as 35.233.39.239" function=tagNode pkg=kubeip

Notice that:

• the Replaced IP for message is here
• the first failing call is one the Google Compute API waiting for the AddAccessConfig operation to complete
• the second failing call is one to the kubernetes API to retrieve nodes in GetNodeByIP (not sure why this one failed)
• the tagging occurred

First conclusion here is that enabling Private Google Access does not resolve the issue completely.
More testing needed...

Note: for subsequent test runs, scaling the node pool from 1 node to 0 then back to 1 is enough.

[avivl]

@pdecat Thanks for the detailed description.
If KubeIP will recover and fix the address and the TAG do you still think that this is an issue?

[pdecat]

It depends,I believe handling the errors and retrying early could improve the situation in that nodes would not stay too long without IP address or tag. When/if the node is left without a public IP, it can no longer report to the master, and when that lasts too long, it is automatically replaced with the auto-repair feature of GKE. In the mean time, I thought of other potential improvements : - Start job with node anti-affinity to perform the updates (problem with single node) - Start Cloud function to perform the updates - Query metadata to track access config updates instead of Compute API Note: supporting the single node scenario is probably not a primary goal of kubeip so it should not block the solutions Then I came across https://github.com/vdm/mig-static-egress-ip/blob/master/README.md which also mentions issues without Private Google Access.

…

[avivl]

@pdecat can you check if the code in https://github.com/doitintl/kubeip/tree/issue-34 solves the problem?

[pdecat]

Hi @avivl,

I've already tried that, and it failed stating the natIP field is immutable or something like that.

An instance can have only one external IP address. If the instance already has an external IP address, you must remove that address first by deleting the old access configuration. Then, you can add a new access configuration with the new external IP address.

source: https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address#IP_assign

[avivl]

Thanks @pdecat
Since KubIP is monitoring a specific node pool " KUBEIP_NODEPOO" if you install kubeip on another non monitored node-pool then we should n't have this problem. WDYT?

[pdecat]

That's indeed a workaround.

[avivl]

Ok so I'll add it to th readme file