Network misbehavior

[liqotech]

Network misbehavior

[alacuku]

Well, this bug does not seem to be related to liqonet but how the kindnet manages the routes between the nodes. It expects the nodes to be in the same L3 network. Based on the logs the PodCIDR is 10.200.0.0/16. It is divided in subnets with netmask 255.255.255.0. Each new subnet is allocated to a node. Each pod deployed an on a given node takes an IP from the subnet assigned to the same node. In order for the pods living in different nodes to communicate between them KindNet configures the routing table. Being the nodes on the same L3 network every thing work smothly.
When we create the virtual node in this case liqo-3dd6f45d-b53a-4f67-8416-505c4d5ddba5 with subnet 10.200.3.0/24 taken by the PodCIDR 10.200.0.0/16, the KindNet running on the other nodes wants to make this subnet reachable by configuring a route that sends the traffic to the virtual node. The liqo-3dd6f45d-b53a-4f67-8416-505c4d5ddba5 is a pod, running on a node, and to it is given an IP taken by the PodCIDR. From the logs we see that the virtual node has IP 10.200.1.14. Looking at the source code of KindNet (https://github.com/kubernetes-sigs/kind/blob/d7f948dd8c00084d6ee30eb953471ce3ce375455/images/kindnetd/cmd/kindnetd/routes.go#L36) it is trying to insert a route for the subnet 10.200.3.0/24 using as gateway the node IP that is 10.200.1.14. It does not succeeds because the virtual node is not directly reachable from the nodes except for the node where the deployment of the virtualkubelet is running.
To solve this problem for KindNet it would be enough to remove the PodCIDR from the node specs of the virtual node.

[palexster]

Thanks a lot @alacuku. I implemented a simple patch as you suggested (#311) and it worked fine. It has to be checked if this does not break compatibility with other CNIs.

[NominalTrajectory]

Hi there! I think I'm experiencing the exact same issue today testing Liqo.

I'm following a quickstart guide described here https://doc.liqo.io/gettingstarted/helloworld/ and using the exact same configurations provided using kind v0.11.1 go1.16.4 darwin/amd64 with Docker Desktop on Mac.

Liqo installs successfully, but as soon as the virtual-kubelet gets created, I see crashing kindnet and kube-proxy pods.

My kindnet logs are:

I0204 12:36:30.478346       1 main.go:316] probe TCP address cluster1-control-plane:6443
I0204 12:36:30.480330       1 main.go:102] connected to apiserver: https://cluster1-control-plane:6443
I0204 12:36:30.480369       1 main.go:107] hostIP = 172.19.0.3
podIP = 172.19.0.3
I0204 12:36:30.480601       1 main.go:116] setting mtu 1500 for CNI
I0204 12:36:30.480640       1 main.go:146] kindnetd IP family: "ipv4"
I0204 12:36:30.480666       1 main.go:150] noMask IPv4 subnets: [10.200.0.0/16]
I0204 12:36:31.075989       1 main.go:223] Handling node with IPs: map[172.19.0.3:{}]
I0204 12:36:31.076049       1 main.go:227] handling current node
I0204 12:36:31.078393       1 main.go:223] Handling node with IPs: map[172.19.0.2:{}]
I0204 12:36:31.078506       1 main.go:250] Node cluster1-worker has CIDR [10.200.1.0/24]
I0204 12:36:31.078776       1 main.go:223] Handling node with IPs: map[10.200.1.9:{}]
I0204 12:36:31.078816       1 main.go:250] Node liqo-b761b1b3-d06c-46a3-94a0-a025c0d2f147 has CIDR [10.200.2.0/24]
I0204 12:36:31.078941       1 routes.go:46] Adding route {Ifindex: 0 Dst: 10.200.2.0/24 Src: <nil> Gw: 10.200.1.9 Flags: [] Table: 0}
I0204 12:36:31.079222       1 main.go:204] Failed to reconcile routes, retrying after error: network is unreachable
I0204 12:36:31.079427       1 main.go:223] Handling node with IPs: map[172.19.0.3:{}]
I0204 12:36:31.079493       1 main.go:227] handling current node
I0204 12:36:31.079511       1 main.go:223] Handling node with IPs: map[172.19.0.2:{}]
I0204 12:36:31.079762       1 main.go:250] Node cluster1-worker has CIDR [10.200.1.0/24]
I0204 12:36:31.080011       1 main.go:223] Handling node with IPs: map[10.200.1.9:{}]
I0204 12:36:31.080094       1 main.go:250] Node liqo-b761b1b3-d06c-46a3-94a0-a025c0d2f147 has CIDR [10.200.2.0/24]
I0204 12:36:31.080244       1 routes.go:46] Adding route {Ifindex: 0 Dst: 10.200.2.0/24 Src: <nil> Gw: 10.200.1.9 Flags: [] Table: 0}
I0204 12:36:31.080403       1 main.go:204] Failed to reconcile routes, retrying after error: network is unreachable
I0204 12:36:32.080503       1 main.go:223] Handling node with IPs: map[172.19.0.3:{}]
I0204 12:36:32.080603       1 main.go:227] handling current node
I0204 12:36:32.080628       1 main.go:223] Handling node with IPs: map[172.19.0.2:{}]
I0204 12:36:32.080651       1 main.go:250] Node cluster1-worker has CIDR [10.200.1.0/24]
I0204 12:36:32.080802       1 main.go:223] Handling node with IPs: map[10.200.1.9:{}]
I0204 12:36:32.080827       1 main.go:250] Node liqo-b761b1b3-d06c-46a3-94a0-a025c0d2f147 has CIDR [10.200.2.0/24]
I0204 12:36:32.081020       1 routes.go:46] Adding route {Ifindex: 0 Dst: 10.200.2.0/24 Src: <nil> Gw: 10.200.1.9 Flags: [] Table: 0}
I0204 12:36:32.081097       1 main.go:204] Failed to reconcile routes, retrying after error: network is unreachable
I0204 12:36:34.081567       1 main.go:223] Handling node with IPs: map[172.19.0.3:{}]
I0204 12:36:34.081681       1 main.go:227] handling current node
I0204 12:36:34.081715       1 main.go:223] Handling node with IPs: map[172.19.0.2:{}]
I0204 12:36:34.081740       1 main.go:250] Node cluster1-worker has CIDR [10.200.1.0/24]
I0204 12:36:34.081976       1 main.go:223] Handling node with IPs: map[10.200.1.9:{}]
I0204 12:36:34.082001       1 main.go:250] Node liqo-b761b1b3-d06c-46a3-94a0-a025c0d2f147 has CIDR [10.200.2.0/24]
I0204 12:36:34.082069       1 routes.go:46] Adding route {Ifindex: 0 Dst: 10.200.2.0/24 Src: <nil> Gw: 10.200.1.9 Flags: [] Table: 0}
I0204 12:36:34.082150       1 main.go:204] Failed to reconcile routes, retrying after error: network is unreachable
I0204 12:36:37.082937       1 main.go:223] Handling node with IPs: map[172.19.0.3:{}]
I0204 12:36:37.083409       1 main.go:227] handling current node
I0204 12:36:37.083493       1 main.go:223] Handling node with IPs: map[172.19.0.2:{}]
I0204 12:36:37.083546       1 main.go:250] Node cluster1-worker has CIDR [10.200.1.0/24]
I0204 12:36:37.083698       1 main.go:223] Handling node with IPs: map[10.200.1.9:{}]
I0204 12:36:37.083771       1 main.go:250] Node liqo-b761b1b3-d06c-46a3-94a0-a025c0d2f147 has CIDR [10.200.2.0/24]
I0204 12:36:37.083964       1 routes.go:46] Adding route {Ifindex: 0 Dst: 10.200.2.0/24 Src: <nil> Gw: 10.200.1.9 Flags: [] Table: 0}
I0204 12:36:37.084140       1 main.go:204] Failed to reconcile routes, retrying after error: network is unreachable
panic: Maximum retries reconciling node routes: network is unreachable

goroutine 1 [running]:
main.main()
	/go/src/cmd/kindnetd/main.go:208 +0x1327

To solve this problem for KindNet it would be enough to remove the PodCIDR from the node specs of the virtual node.

I tried this but it didn't seem to work for me.

@palexster can this issue be reopened?