Use Workload Identityfor KubeIP

[doitintl]

Use Workload Identityfor KubeIP

[jawabuu]

Hey @alexei-led Any progress on this?
I tried using workload identity and got this

time="2020-12-07T14:23:12Z" level=info msg="&{kubeip static-ingress ingress-pool true [] 5ns false}"
time="2020-12-07T14:23:12Z" level=info msg="[]"
time="2020-12-07T14:23:12Z" level=info msg="kubeIP is starting" Build Date="2020-10-21T14:08:36+0000" Cluster name=kluster Project name=project-gc Version=v0
time="2020-12-07T14:23:12Z" level=info msg="Starting kubeip controller" pkg=kubeip-node
time="2020-12-07T14:23:12Z" level=info msg="Starting forceAssignment" function=forceAssignment pkg=kubeip
time="2020-12-07T14:23:12Z" level=info msg="Collecting Node List..." function=processAllNodes pkg=kubeip
time="2020-12-07T14:23:12Z" level=info msg="kubeip controller synced and ready" pkg=kubeip-node
time="2020-12-07T14:23:14Z" level=error msg="Get \"https://www.googleapis.com/compute/v1/projects/project-gc/regions/us-central1/addresses?alt=json&filter=%28status%3DIN_USE%29+AND+%28labels.kubeip%3Dstatic-ingress%29\": Get \"http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token\": net/http: timeout awaiting response headers"
time="2020-12-07T14:23:14Z" level=info msg="Found unassigned node gke-kluster-ingress-pool-71deba9d-c27d in pool ingress-pool" function=processAllNodes pkg=kubeip
time="2020-12-07T14:23:14Z" level=info msg="Working on gke-kluster-ingress-pool-71deba9d-c27d in zone us-central1-a" function=Kubeip pkg=kubeip
time="2020-12-07T14:23:16Z" level=error msg="Get \"https://www.googleapis.com/compute/v1/projects/project-gc/regions/us-central1/addresses?alt=json&filter=%28status%3DIN_USE%29+AND+%28labels.kubeip%3Dstatic-ingress%29\": metadata: GCE metadata \"instance/service-accounts/default/token\" not defined"
time="2020-12-07T14:23:16Z" level=info msg="Found unassigned node gke-kluster-ingress-pool-71deba9d-l13z in pool ingress-pool" function=processAllNodes pkg=kubeip
time="2020-12-07T14:23:16Z" level=error msg="Failed to list IP addresses in region us-central1: \"Get \\\"https://www.googleapis.com/compute/v1/projects/project-gc/regions/us-central1/addresses?alt=json&filter=%28status%3DRESERVED%29+AND+%28labels.kubeip%3Dstatic-ingress%29+AND++%28-labels.kubeip-node-pool%3A%2A%29\\\": metadata: GCE metadata \\\"instance/service-accounts/default/token\\\" not defined\"" function=findFreeAddress pkg=kubeip
time="2020-12-07T14:23:16Z" level=info msg="Get \"https://www.googleapis.com/compute/v1/projects/project-gc/regions/us-central1/addresses?alt=json&filter=%!s(MISSING)tatus%!D(MISSING)RESERVED%!+(MISSING)AND+%!l(MISSING)abels.kubeip%!D(MISSING)static-ingress%!+(MISSING)AND++%!-(MISSING)labels.kubeip-node-pool%!A(MISSING)%!A(MISSING)%!\"(MISSING): metadata: GCE metadata \"instance/service-accounts/default/token\" not defined"

[alexei-led]

@jawabuu will check this, thank you for reporting

[jawabuu]

Thanks @alexei-led I'm ready to test whenever you have a build

[jawabuu]

Hey @alexei-led I managed to get this working with workload identity. Feel free to close the issue.
A bit of documentation should suffice for future users.

[alexei-led]

@jawabuu glad to hear you worked it out

[TeunKooijman]

@jawabuu would you mind sharing how you got this to work? I'm struggling with this myself, and I'm getting the exact same results as you've posted here. Thanks in advance for any help!

[jawabuu]

Hey @TeunKooijman are you comfortable using terraform?
If so here's how I to do it
https://github.com/nufailtd/terraform-budget-gcp/blob/main/modules/kubeip
Specifically

module "kubeip" {
  source = "github.com/nufailtd/terraform-budget-gcp//modules/kubeip"
  project_id             = project_id
  zone                   = gke_location
  host                   = gke_host
  cluster_ca_certificate = gke_cluster_ca_certificate
  token                  = gke_token
}

[jawabuu]

The root cause for me was that the kubernetes service account linked to the google service account was not mounted to the kubeip pod.

[TeunKooijman]

Thanks for getting back to me @jawabuu. As a matter of fact I too am using Terraform (in part). A couple of things already stood out from looking at your implementation:

• I was using google_project_iam_binding where you were using google_project_iam_member, might have just been me getting lost in translation somewhere.
• You are using automountServiceAccountToken: true, which if I understood correctly is the default on the latest version, but I've added it regardless.
• You are using Traefik where I am using Nginx as an ingress controller, but it should work reasonably similar.

See https://github.com/silvester-io/gcp-bootstrap/blob/master/terraform/kubeip/kubeip.tf for the current infrastructure.

The ingress node, as well as the nginx ingress controller on it runs fine, until I apply the kubeip-as-a-helm-chart. That's when the ingress node's status becomes Unknown on the GCP cluster dashboard, and the nginx workload starts reporting Kubelet stopped posting node status.

Did you ever run into anything like this?

[jawabuu]

Hey @TeunKooijman I am going through your code.
What is the name of your kubernetes service account? Are you annotating your kubernetes service account?

[TeunKooijman]

Hi @jawabuu, thanks a lot for going through all this trouble to help me out. The service account (both the k8s and gcp one) should be called kubeip-serviceaccount. As for the annotation, see line 6 on: https://github.com/silvester-io/gcp-bootstrap/blob/master/charts/kubeip/templates/service-account.yaml

The results of helm template ./charts/kubeip --namespace kube-system -f ./charts/kubeip/values.yaml is:

# Source: kubeip/templates/service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    iam.gke.io/gcp-service-account: [email protected]
  name: kubeip-serviceaccount
  namespace: kube-system
automountServiceAccountToken: true
---
# Source: kubeip/templates/service-account.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubeip-serviceaccount
  namespace: kube-system
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
# Source: kubeip/templates/service-account.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubeip-serviceaccount
subjects:
  - kind: ServiceAccount
    name: kubeip-serviceaccount
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: kubeip-serviceaccount
  apiGroup: rbac.authorization.k8s.io
---
# Source: kubeip/templates/config-map.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    app: kubeip
  name: kubeip-config
  namespace: kube-system
data:
  KUBEIP_LABELKEY: "kubeip"
  KUBEIP_LABELVALUE: "silvester-cluster"
  KUBEIP_NODEPOOL: "silvester-nodepool-ingress"
  KUBEIP_FORCEASSIGNMENT: "true"
  KUBEIP_ADDITIONALNODEPOOLS: ""
  KUBEIP_TICKER: "5"
  KUBEIP_ALLNODEPOOLS: "false"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kubeip
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kubeip
  template:
    metadata:
      labels:
        app: kubeip
    spec:
      priorityClassName: system-cluster-critical
      nodeSelector:
        cloud.google.com/gke-nodepool:  silvester-nodepool-apps
      containers:
        - name: "kubeip"
          image: doitintl/kubeip:latest
          imagePullPolicy: Always
          env:
            - name: "KUBEIP_LABELKEY"
              valueFrom:
                configMapKeyRef:
                  key: "KUBEIP_LABELKEY"
                  name: kubeip-config
            - name: "KUBEIP_LABELVALUE"
              valueFrom:
                configMapKeyRef:
                  key: "KUBEIP_LABELVALUE"
                  name: kubeip-config
            - name: "KUBEIP_NODEPOOL"
              valueFrom:
                configMapKeyRef:
                  key: "KUBEIP_NODEPOOL"
                  name: kubeip-config
            - name: "KUBEIP_FORCEASSIGNMENT"
              valueFrom:
                configMapKeyRef:
                  key: "KUBEIP_FORCEASSIGNMENT"
                  name: kubeip-config
            - name: "KUBEIP_ADDITIONALNODEPOOLS"
              valueFrom:
                configMapKeyRef:
                  key: "KUBEIP_ADDITIONALNODEPOOLS"
                  name: kubeip-config
            - name: "KUBEIP_TICKER"
              valueFrom:
                configMapKeyRef:
                  key: "KUBEIP_TICKER"
                  name: kubeip-config
            - name: "KUBEIP_ALLNODEPOOLS"
              valueFrom:
                configMapKeyRef:
                  key: "KUBEIP_ALLNODEPOOLS"
                  name: kubeip-config
      restartPolicy: Always
      serviceAccountName: kubeip-serviceaccount
      automountServiceAccountToken: true

[jawabuu]

No problem @TeunKooijman
Configmap seems correct
I guess what's left is to actually test that a pod with that service account mounted can perform the required API operations

Please try this

module "test-workload-identity" {
  source = "github.com/nufailtd/terraform-budget-gcp//modules/test-workload-identity"

  host                   = gke_auth.host
  cluster_ca_certificate = gke_cluster_ca_certificate
  token                  = gke_token
  ksa                    = k8s_service_account_name
  ksa_namespace          = k8s_service_account_namespace
  run_post_install       = true
}

[jawabuu]

Then exec into the pod, and
gcloud auth list

[TeunKooijman]

The source URL gives me a 404 (even when removing the duplicate /), but if you're talking about step 7 of the workload identity (exec'ing into the pod and testing gcloud auth list) that works for:

apiVersion: v1
kind: Pod
metadata:
  name: workload-identity-test
  namespace: K8S_NAMESPACE
spec:
  containers:
  - image: google/cloud-sdk:slim
    name: workload-identity-test
    command: ["sleep","infinity"]
  serviceAccountName: KSA_NAME

I must admit that I last checked this a couple of days ago, so I'll give it another try.

[TeunKooijman]

workload-identity-test.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: workload-identity-test
  namespace: kube-system
spec:
  containers:
  - image: google/cloud-sdk:slim
    name: workload-identity-test
    command: ["sleep","infinity"]
  serviceAccountName: kubeip-serviceaccount

kubectl apply -f workload-identity-test.yaml
kubectl exec --stdin --tty -n kube-system workload-identity-test -- /bin/bash
gcloud auth list

result is:

root@workload-identity-test:/# gcloud auth list
                        Credentialed Accounts
ACTIVE  ACCOUNT
*       [email protected]

To set the active account, run:
    $ gcloud config set account `ACCOUNT`

root@workload-identity-test:/#

Does that seem right?

[jawabuu]

That's perfect.
Could you try retrieving a token?

[TeunKooijman]

The results of root@workload-identity-test:/# gcloud container clusters get-credentials silvester-cluster --region europe-west1-b are:

WARNING: Accessing a Kubernetes Engine cluster requires the kubernetes commandline
client [kubectl]. To install, run
  $ gcloud components install kubectl

Fetching cluster endpoint and auth data.
kubeconfig entry generated for silvester-cluster.

Is this what you meant with retrieving a token?

[jawabuu]

Once you exec inside the test workload identity
pod, try retrieving an access token. Let me see if I can find an example.

[jawabuu]

Try a combination of these

url=''
token=$(curl -vv -H "Metadata-Flavor: Google"  http://metadata/computeMetadata/v1/instance/service-accounts/default/identity?audience=$url)
curl -vv -H "Authorization: Bearer $token" $url
curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" url

url is the google api url that you will call to for example list Instances, IPs e.t.c

[TeunKooijman]

Do you perhaps have an example for a URL?

[TeunKooijman]

I think I may have already figured it out. This is the content of the retrieved token (without header and signature):

{
  "aud": "http://metadata.google.internal/computeMetadata/v1/instance/disks",
  "azp": "117123956372577273588",
  "email": "[email protected]",
  "email_verified": true,
  "exp": 1614038551,
  "iat": 1614034951,
  "iss": "https://accounts.google.com",
  "sub": "117123956372577273588"
}

[jawabuu]

You could also share the debug logs for your KubeIP deployment

[jawabuu]

In my logs, I had an error with a url shuch as this
https://www.googleapis.com/compute/v1/projects/project-gc/regions/us-central1/addresses?alt=json
You will need to substitute for your project.

[jawabuu]

I think I may have already figured it out. This is the content of the retrieved token (without header and signature):

{
  "aud": "http://metadata.google.internal/computeMetadata/v1/instance/disks",
  "azp": "117123956372577273588",
  "email": "[email protected]",
  "email_verified": true,
  "exp": 1614038551,
  "iat": 1614034951,
  "iss": "https://accounts.google.com",
  "sub": "117123956372577273588"
}

I see

[TeunKooijman]

kubeip pod logs look like this:

time="2021-02-22T20:31:59Z" level=info msg="&{kubeip silvester-cluster silvester-nodepool-ingress true [] 5ns false}"
time="2021-02-22T20:31:59Z" level=info msg="[]"
time="2021-02-22T20:31:59Z" level=info msg="kubeIP is starting" Build Date="2020-10-21T14:08:36+0000" Cluster name=silvester-cluster Project name=silvester-304916 Version=v0
time="2021-02-22T20:31:59Z" level=info msg="Starting forceAssignment" function=forceAssignment pkg=kubeip
time="2021-02-22T20:31:59Z" level=info msg="Starting kubeip controller" pkg=kubeip-node
time="2021-02-22T20:31:59Z" level=info msg="Collecting Node List..." function=processAllNodes pkg=kubeip
time="2021-02-22T20:31:59Z" level=info msg="kubeip controller synced and ready" pkg=kubeip-node
time="2021-02-22T20:32:01Z" level=error msg="Get \"https://www.googleapis.com/compute/v1/projects/silvester-304916/regions/europe-west1/addresses?alt=json&filter=%28status%3DIN_USE%29+AND+%28labels.kubeip%3Dsilvester-cluster%29\": Get \"http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token\": net/http: timeout awaiting response headers"
time="2021-02-22T20:32:01Z" level=info msg="Found unassigned node gke-silvester-cluste-silvester-nodepo-f3c75d49-wj20 in pool silvester-nodepool-ingress" function=processAllNodes pkg=kubeip
time="2021-02-22T20:32:01Z" level=info msg="Working on gke-silvester-cluste-silvester-nodepo-f3c75d49-wj20 in zone europe-west1-b" function=Kubeip pkg=kubeip
time="2021-02-22T20:32:01Z" level=info msg="Found node without tag gke-silvester-cluste-silvester-nodepo-f3c75d49-wj20" function=assignMissingTags pkg=kubeip
time="2021-02-22T20:32:02Z" level=info msg="Found reserved address 35.210.199.226" function=replaceIP pkg=kubeip
time="2021-02-22T20:32:14Z" level=info msg="Replaced IP for gke-silvester-cluste-silvester-nodepo-f3c75d49-wj20 zone europe-west1-b new ip 35.210.199.226" function=replaceIP pkg=kubeip
time="2021-02-22T20:32:14Z" level=info msg="Tagging node gke-silvester-cluste-silvester-nodepo-f3c75d49-wj20 as 35.210.199.226" function=tagNode pkg=kubeip
time="2021-02-22T20:36:59Z" level=info msg="On Ticker" function=forceAssignment pkg=kubeip

The following part then just repeats every 4 or 5 minutes-ish:

time="2021-02-22T20:41:59Z" level=info msg="On Ticker" function=forceAssignment pkg=kubeip
time="2021-02-22T20:41:59Z" level=info msg="Collecting Node List..." function=processAllNodes pkg=kubeip
time="2021-02-22T20:41:59Z" level=info msg="Found unassigned node gke-silvester-cluste-silvester-nodepo-f3c75d49-wj20 in pool silvester-nodepool-ingress" function=processAllNodes pkg=kubeip
time="2021-02-22T20:41:59Z" level=info msg="Working on gke-silvester-cluste-silvester-nodepo-f3c75d49-wj20 in zone europe-west1-b" function=Kubeip pkg=kubeip
time="2021-02-22T20:42:00Z" level=info msg="Found reserved address 35.210.199.226" function=replaceIP pkg=kubeip
time="2021-02-22T20:42:02Z" level=info msg="Replaced IP for gke-silvester-cluste-silvester-nodepo-f3c75d49-wj20 zone europe-west1-b new ip 35.210.199.226" function=replaceIP pkg=kubeip
time="2021-02-22T20:42:02Z" level=info msg="Tagging node gke-silvester-cluste-silvester-nodepo-f3c75d49-wj20 as 0.0.0.0" function=tagNode pkg=kubeip
time="2021-02-22T20:42:02Z" level=info msg="Tagging node gke-silvester-cluste-silvester-nodepo-f3c75d49-wj20 as 35.210.199.226" function=tagNode pkg=kubeip

[jawabuu]

And is the IP assigned to the node successfully?

[TeunKooijman]

According to https://console.cloud.google.com/networking/addresses/list?project=silvester-304916 it is not

[TeunKooijman]

And according to kubectl get nodes -w -o wide I don't think either:

NAME                                                  STATUS     ROLES    AGE     VERSION             INTERNAL-IP     EXTERNAL-IP      OS-IMAGE                             KERNEL-VERSION   CONTAINER-RUNTIME
gke-silvester-cluste-silvester-nodepo-d99cae24-gdhm   Ready      <none>   3h41m   v1.18.12-gke.1210   10.132.15.215   35.233.92.127    Container-Optimized OS from Google   5.4.49+          docker://19.3.9
gke-silvester-cluste-silvester-nodepo-f3c75d49-wj20   NotReady   <none>   161m    v1.18.12-gke.1210   10.132.15.218   35.241.157.151   Container-Optimized OS from Google   5.4.49+          docker://19.3.9

[jawabuu]

How many nodepools do you have?

[TeunKooijman]

Two, silvester-nodepool-apps and silvester-nodepool-ingress.

[jawabuu]

Is there a particular reason the 2nd node is NotReady?

[TeunKooijman]

It switched from Ready to Unknown (on GCP dashboard) and NotReady via kubectl describe as soon as I apply the kubeip deployment.

[jawabuu]

Which nodepool does it belong to?

[TeunKooijman]

It belongs to silvester-nodepool-ingres, the one that kubeip tries to assign the external IP on. kubeip does this from the silvester-nodepool-apps node pool node.

[TeunKooijman]

Through kubectl describe node gke-silvester-cluste-silvester-nodepo-f3c75d49-wj20 it even shows the kubip_assigned=35-210-199-226 label on the node.

[jawabuu]

Can you terraform modules allow me to reproduce the issue?

[TeunKooijman]

I'm afraid we're treading into unfamiliar territory for me. This is the first time I've done anything with Terraform. I assume you're looking for a way to lift-and-shift this to a GCP project that you control? I could look into how that's done with a Terraform module, but that would have to be tomorrow, as it's sadly getting too late for me for tonight. The other option would be to fork https://github.com/silvester-io/gcp-bootstrap and change the references to the GCP project that are (sadly) currently a bit spread around:

/terraform/kubernetes/variables.tf
/terraform/kubeip/variables.tf
/terraform/cluster/variables.tf
/charts/kubeip/values.yaml
/.github/workflows/main.yml

And subsequently add a GCP_SA_KEY and run the GitHub action.

[jawabuu]

I'll check it out and see if I can help.

[TeunKooijman]

That would be amazing, thanks a lot @jawabuu. Just a little heads up that I'm currently manually installing the kubeip after the github action manually (see the commented out helm install of kubeip in the bottom of the GitHub Action). What you'll (hopefully) see, is that the ingress node pool node is ready, but then as soon as kubeip does it's thing it switches to Unknown/NotReady.

[jawabuu]

Hey @TeunKooijman I was unable to create a cluster using your terraform file.
I however was able to use the terraform gke module and run KubeIP successfully. Do you mind adapting your code to use the the terraform gke module?

[TeunKooijman]

That wouldn't be a problem at all. Do you perhaps have an example somewhere of what you did or how you used the gke terraform module?

[jawabuu]

I will push my changes to a fork shortly.

[jawabuu]

Hey @TeunKooijman Try the example at https://github.com/jawabuu/gcp-bootstrap

[TeunKooijman]

Hi @jawabuu, took me a bit of time, but I've set it up with the gke module. However, the exact same thing is happening. The kubeip pod just repeats the following again every 5 minutes in the pod:

time="2021-02-25T16:59:21Z" level=info msg="On Ticker" function=forceAssignment pkg=kubeip
time="2021-02-25T16:59:21Z" level=info msg="Collecting Node List..." function=processAllNodes pkg=kubeip
time="2021-02-25T16:59:22Z" level=info msg="Found unassigned node gke-silvester-cluster-ingress-pool-9728e76b-w1qq in pool ingress-pool" function=processAllNodes pkg=kubeip
time="2021-02-25T16:59:22Z" level=info msg="Working on gke-silvester-cluster-ingress-pool-9728e76b-w1qq in zone europe-west1-b" function=Kubeip pkg=kubeip
time="2021-02-25T16:59:22Z" level=info msg="Found reserved address 35.210.199.226" function=replaceIP pkg=kubeip
time="2021-02-25T16:59:25Z" level=info msg="Replaced IP for gke-silvester-cluster-ingress-pool-9728e76b-w1qq zone europe-west1-b new ip 35.210.199.226" function=replaceIP pkg=kubeip
time="2021-02-25T16:59:25Z" level=info msg="Tagging node gke-silvester-cluster-ingress-pool-9728e76b-w1qq as 0.0.0.0" function=tagNode pkg=kubeip
time="2021-02-25T16:59:25Z" level=info msg="Tagging node gke-silvester-cluster-ingress-pool-9728e76b-w1qq as 35.210.199.226" function=tagNode pkg=kubeip

• The ingress-pool is status is "Unknown" (on the GCP portal), and NotReady (in kubectl).
• External IP is not "in use" by anything.
• The ingress pool does have the kubip_assigned=35-210-199-226 label.
• I've also moved the nginx-ingress-controller from Helm to Terraform, but even before I installed that, and it was just kubeip running, the status already became unknown.

However, now that I've moved away from Helm entirely, you should be able to use the Terraform code in my repository now, as it's currently just a matter of terraform init & apply.

Any ideas left?

[jawabuu]

Hey @TeunKooijman Sorry to hear that.
Can you try out my code from your local workstation?
Or better yet in Google Cloud Shell
https://github.com/jawabuu/gcp-bootstrap

1. Destroy any existing terraform deployments
2. cd into terraform/cluster
3. Run gcloud auth login
4. Run . ./auth.sh [your_project_name]
5. terraform init
6. terraform plan
7. terraform apply

[jawabuu]

time="2021-02-25T16:59:25Z" level=info msg="Tagging node gke-silvester-cluster-ingress-pool-9728e76b-w1qq as 0.0.0.0" function=tagNode pkg=kubeip

I also don't think this is right
Tagging as 0.0.0.0

[TeunKooijman]

I could do that, but I must confess that I do not fully understand what the auth.sh script does, and that I feel a little bit weary running a script that does something that I don't understand. Could you perhaps give me a slight explanation of what it does and why that's needed?

[jawabuu]

It sets these variables for use by Terraform.

export GOOGLE_IMPERSONATE_SERVICE_ACCOUNT=$SERVICE_ACCOUNT
export GOOGLE_OAUTH_ACCESS_TOKEN=$(gcloud auth print-access-token)

You can skip it.

[TeunKooijman]

Hi @jawabuu, I couldn't get your terraform files to work, sadly, so I've done some more digging on my own. It turns out that when I SSH into the ingress-pool node and query the metadata endpoint for the current access-config:

curl -i -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/network-interfaces/0/access-configs/0/external-ip

The ip-address is actually completely empty.
Trying to manually set the external ip address through:

gcloud compute instances add-access-config gke-silvester-cluster-ingress-pool-01e60f2e-k2z8 --zone=europe-west1-b --address=35.210.***.***

Yields a clear error message, saying:

ERROR: (gcloud.compute.instances.add-access-config) Could not fetch resource:
 - External IP address: 35.210.***.*** has a different network tier STANDARD from the network tier in instance access config PREMIUM.

I define the IP address as STANDARD tier on line 7 of: https://github.com/silvester-io/gcp-bootstrap/blob/master/terraform/kubeip/kubeip.tf but the compute instance node's network (or the terraform-google-modules/network/google?) is probably defined as PREMIUM by default.

My hope is that once I figure out how the change that the STANDARD as well, things will start working more smoothly.

[TeunKooijman]

Yep, it looks like that was it. Adding the following to Terraform allowed kubeip to do what it had to do:

resource "google_compute_project_default_network_tier" "default" {
  network_tier = "STANDARD"
}

@alexei-led not sure if this is anything you want to do something with (as kubeip currently does not log any errors when this happens), but perhaps just mentioning on the readme.md that if the ingress node becomes Unknown/NotReady, it might be because the network_tier and ip_address tier differ.

Anyway, @jawabuu, thanks a bunch for thinking along with me!

[jawabuu]

Hey @TeunKooijman
Glad you got to solve it.