kube-vip works for a few seconds and then stops functioning

[kube-vip]

kube-vip works for a few seconds and then stops functioning

[thebsdbox]

The current release on the releases page is 0.1.8 where 0.1.9 is in active development (which to read between the lines, means i've broken it) 👎 The two arp PRs have introduced some behaviour that is being investigated.

[jjschwarz]

OK, my fault on that...didn't realize 0.1.9 wasn't a release. I will retry with 0.1.8

[jjschwarz]

That resolved the issue. Thank you!

[thebsdbox]

I'll close this and put process in place to make sure we test differently to save confusion.

[dhawal55]

@thebsdbox Is this resolved in the v0.1.9 release? I tried with v0.1.9 and facing the same issue. The VIP IP gets registered first but stops working after a while. I notice the Kube-vip pod crashes with below error and the new pod acquires leadership but somehow the IP doesn't get registered under the specified interface:

Old kube-vip pod:

time=“2020-10-26T19:45:08Z” level=info msg=“Broadcasting ARP update for 10.17.40.224 (00:50:56:a1:4b:54) via eth0"
time=“2020-10-26T19:45:11Z” level=info msg=“Broadcasting ARP update for 10.17.40.224 (00:50:56:a1:4b:54) via eth0"
time=“2020-10-26T19:45:14Z” level=info msg=“Broadcasting ARP update for 10.17.40.224 (00:50:56:a1:4b:54) via eth0"
E1026 19:45:15.096740       1 leaderelection.go:321] error retrieving resource lock kube-system/plunder-lock: Get “https://dpatel-test-mxrh4:6443/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/plunder-lock”: context deadline exceeded
I1026 19:45:15.096893       1 leaderelection.go:277] failed to renew lease kube-system/plunder-lock: timed out waiting for the condition
E1026 19:45:15.097056       1 leaderelection.go:297] Failed to release lock: resource name may not be empty
time=“2020-10-26T19:45:15Z” level=info msg=“This node is becoming a follower within the cluster”
time=“2020-10-26T19:45:15Z” level=info msg=“Shutting down Kube-Vip Leader Election cluster”

New kube-vip pod:

time="2020-10-26T19:45:26Z" level=info msg="Beginning cluster membership, namespace [kube-system], lock name [plunder-lock], id [dpatel-test-mxrh4]"
I1026 19:45:26.707350       1 leaderelection.go:242] attempting to acquire leader lease  kube-system/plunder-lock...
E1026 19:45:39.790519       1 leaderelection.go:321] error retrieving resource lock kube-system/plunder-lock: etcdserver: request timed out
E1026 19:45:50.506892       1 leaderelection.go:321] error retrieving resource lock kube-system/plunder-lock: etcdserver: request timed out
I1026 19:45:52.710399       1 leaderelection.go:252] successfully acquired lease kube-system/plunder-lock
time="2020-10-26T19:45:52Z" level=info msg="Node [dpatel-test-mxrh4] is assuming leadership of the cluster"
time="2020-10-26T19:45:52Z" level=info msg="This node is starting with leadership of the cluster"
time="2020-10-26T19:45:52Z" level=info msg="Broadcasting ARP update for 10.17.40.224 (00:50:56:a1:4b:54) via eth0"
time="2020-10-26T19:45:55Z" level=info msg="Broadcasting ARP update for 10.17.40.224 (00:50:56:a1:4b:54) via eth0"
time="2020-10-26T19:45:58Z" level=info msg="Broadcasting ARP update for 10.17.40.224 (00:50:56:a1:4b:54) via eth0"
time="2020-10-26T19:46:01Z" level=info msg="Broadcasting ARP update for 10.17.40.224 (00:50:56:a1:4b:54) via eth0"
time="2020-10-26T19:46:04Z" level=info msg="Broadcasting ARP update for 10.17.40.224 (00:50:56:a1:4b:54) via eth0"
time="2020-10-26T19:46:07Z" level=info msg="Broadcasting ARP update for 10.17.40.224 (00:50:56:a1:4b:54) via eth0"
time="2020-10-26T19:46:10Z" level=info msg="Broadcasting ARP update for 10.17.40.224 (00:50:56:a1:4b:54) via eth0"
time="2020-10-26T19:46:14Z" level=info msg="Broadcasting ARP update for 10.17.40.224 (00:50:56:a1:4b:54) via eth0"
time="2020-10-26T19:46:17Z" level=info msg="Broadcasting ARP update for 10.17.40.224 (00:50:56:a1:4b:54) via eth0"

When I restart the kube-vip pod, it works again for a min and then stops working

[thebsdbox]

We've introduced a behaviour that may cause additional failover. We're preparing a fix!

[thebsdbox]

Reopening until the fix is in place.

[thebsdbox]

This is fixed in 0.2.0

[dhawal55]

@thebsdbox Same issue with 0.2.0 so it's not fixed, works with 0.8.0

[thebsdbox]

64 bytes from 192.168.0.40: icmp_seq=101 ttl=64 time=1.493 ms
64 bytes from 192.168.0.40: icmp_seq=102 ttl=64 time=1.403 ms
92 bytes from 192.168.0.41: Redirect Host(New addr: 192.168.0.40)
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 4db0   0 0000  3f  01 abcc 192.168.0.180  192.168.0.40 

Request timeout for icmp_seq 103
92 bytes from 192.168.0.41: Redirect Host(New addr: 192.168.0.40)
Vr HL TOS  Len   ID Flg  off TTL Pro  cks      Src      Dst
 4  5  00 0054 8e3c   0 0000  3f  01 6b40 192.168.0.180  192.168.0.40 

Request timeout for icmp_seq 104
Request timeout for icmp_seq 105
Request timeout for icmp_seq 106
Request timeout for icmp_seq 107
64 bytes from 192.168.0.40: icmp_seq=108 ttl=64 time=1.423 ms
64 bytes from 192.168.0.40: icmp_seq=109 ttl=64 time=1.620 ms

$ kubectl describe pods  kube-vip-k8s01 -n kube-system

...

Containers:
  kube-vip:
    Container ID:  docker://615722853a756df2235abd2c972354c837dc98c3c796601223174774a33b813e
    Image:         plndr/kube-vip:0.2.0
    Image ID:      docker-pullable://plndr/kube-vip@sha256:71b22e427820f57c0d4cd99ccf832e44438e9c6d917a1b5ef9d082055debe9ea
    Port:          <none>
    Host Port:     <none>
    Args:
      start
    State:          Running
      Started:      Mon, 02 Nov 2020 18:11:23 +0000
    Last State:     Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Mon, 02 Nov 2020 18:09:04 +0000
      Finished:     Mon, 02 Nov 2020 18:10:58 +0000
    Ready:          True
    Restart Count:  2
    Environment:
      vip_arp:             true
      vip_interface:       ens160
      vip_leaderelection:  true
      vip_leaseduration:   5
      vip_renewdeadline:   3
      vip_retryperiod:     1
      vip_address:         192.168.0.40
    Mounts:
      /etc/kubernetes/admin.conf from kubeconfig (rw)
      /etc/ssl/certs from ca-certs (ro)