Can Kilo Work With Nodes Behind NAT?

[squat]

Can Kilo Work With Nodes Behind NAT?

[squat]

Hey @gravypod at least one peer in the WireGuard mesh will need to have a public IP and a publicly accessible port, eg if you run all nodes on the edge behind NAT but run the control plane in the cloud. At first, the node in the cloud will not be able to connect to the edge nodes, but once the edge nodes are booted and connect to the cloud node, the IP and port of the edge node will automatically be recorded in the cloud node’s configuration. In particular, you can use a helpful WireGuard configuration feature called PersistentKeepAlive, which instructs WireGuard to periodically send a keep alive packet to keep the port open and I’m this way be able to traverse NAT gateways. In this case WireGuard will work just fine: a node in the cloud has a publicly accessible IP and port and all nodes behind users’ NAT can connect to it. I use this exact configuration in several deployments and it is one of the primary use cases for this project :)

[frbncis]

Do you have any documentation on how to apply the PersistentKeepAlive setting on the edge nodes? I've tried editing /var/lib/kilo/conf but that gets overwritten and I don't see any annotations available to specify the value on a given node.

Many thanks for this project!

[squat]

Hi @frbncis right now the PersistentKeepalive setting can be only be adjusted for Peers added via the Peer custom resource. Currently it cannot be set for nodes that are directly part of the Kubernetes cluster. It would not be difficult to add at all. It could be implemented via another node annotation. Would that be helpful? Let me know and id be happy to write the feature (or review a PR :))

[frbncis]

Out of curiosity, I decided to give a try at implementing here: master...frbncis:node-keepalives

Seems to be working for me, on the node behind the NAT:

interface: kilo0
  public key: L769oKuo5GItkx3hC4oDrfFrHaNf/fF+7NDEJLWfFyQ=
  private key: (hidden)
  listening port: 51820

peer: KIp5hMoLTayQ9XndUvO4ZkBC9d65jLZoGcMWTg+0jGo=
  endpoint: <cloud-node-public-ip>:51820
  allowed ips: 10.42.0.0/24, 10.20.0.5/32, 10.4.0.1/32
  latest handshake: 21 seconds ago
  transfer: 83.57 KiB received, 253.40 KiB sent
  persistent keepalive: every 2 seconds

The cloud node is annotated with kilo.squat.ai/wireguard-persistent-keepalive: 2

If I'm on the right path, I'd be happy iterate on it and put it in as a PR.

[squat]

💯 totally well done! I read through the commit and the only comment I would have is to keep the annotation and struct field name succinct: WireGuardPersistentKeepalive -> PersistentKeepalive

Nicely done :)

[squat]

Tht would be a great feature to document in a short doc. We can add that in a follow up PR of course

[squat]

Fixed by #31 (thanks @frbncis)

Please reopen if you see this doesn't address your use-case

[carlosrmendes]

Hi,
I have one master node in region A with a public ip and a worker node in region B behind a NAT (two separate networks).

After deploy Kilo I annotated both nodes to force external ip (master with own public ip and worker with NAT public ip) and to set the related location on each (master: region-a, worker: region-b).

Checking the wireguard peers in the master, with wg command, I can see the peer of the worker, with the NAT public ip as endpoint, but the port is different than the wireguard listen port set on the worker node.

I can also see that the an handshake was made successfully, but after 30s approximately, the Kilo recreate the peer because it detects differences on configuration (log: 'WireGuard configurations are different'), due to the endpoint port and interrupting existing connections.

How can I solve this?
Thanks in advance.

[squat]

Hi @carlosrmendes let's discuss this in a new issue :) please open a new issue and post the WireGuard configurations for the master and worker so we can see what is going on (we can redact keys and IPs of course).

Thanks,
-Lucas

[eddiewang]

Hi there @carlosrmendes and @squat - were you two ever able to trace this down? I am also seeing an issue where the Wireguard connection is made, but Kilo is overwriting the Endpoint in the config, causing the connection to reset, and the host to be unreachable.

In the Kilo pods, logs read that WireGuard configurations are different.

Is there any way to statically set the static Endpoint in the peer yaml?

[squat]

Hi @eddiewang yes this bug was fixed a few months ago. Your issue sounds a bit different actually. Can you file a new issue and describe your scenario in detail?

[eddiewang]

@squat turns out it has to do with kilo overwriting the endpoint once the connection was established. I manually specified the endpoint in the yaml and things persisted again. although this is running rke with flannel as the primary CNI. I can't seem to get kilo to behave well in CNI mode.

[eddiewang]

Quick update on this. Copying the kubeconfig yml k3s.yml to all slave nodes does seem to work consistently. Not ideal, but it's a workable patch for now.

[squat]

Yes, this is the correct current usage

[squat]

Would be worth commenting here: https://github.com/squat/kilo/blob/master/manifests/kilo-k3s.yaml#L169

[eddiewang]

doing some packing this weekend, but i can make a quick PR for that later this week.

[squat]

Thanks @eddiewang that would be great 💪

[eddiewang]

Done via #61