Labels to differentiate peers

[jodevsa]

Labels to differentiate peers

[jodevsa]

Suppose, that this variant is even better. But it depends on my own flow. Which resources are created first. If there is a chance to use not IP: 8.8.8.8, but namespace: Alice, then it will completely resolve my issue.

Filtering by namespace is a complex task so I'm thinking of delivering this feature in multiple phases to break the work in smaller chunks and make delivery possible

initial plan:

• egress rules for destination IP (Phase 1)

apiVersion: vpn.example.com/v1alpha1
kind: WireguardPeer
metadata:
  name: peer1
spec:
  wireguardRef: "my-cool-vpn"
  networkPolicy:
    egress:
     - action: Allow 
        destination:
            IP: 8.8.8.8

• egress rules based on kubernetes service (Phase 2)

apiVersion: vpn.example.com/v1alpha1
kind: WireguardPeer
metadata:
  name: peer1
spec:
  wireguardRef: "my-cool-vpn"
  networkPolicy:
    egress:
     - action: Allow 
        destination:
            service:
                 service: "service-1"
                 namespace: "A"

• egress rules based on namespace (phase3)

apiVersion: vpn.example.com/v1alpha1
kind: WireguardPeer
metadata:
  name: peer1
spec:
  wireguardRef: "my-cool-vpn"
  networkPolicy:
    egress:
     - action: Allow 
        namespace: "A"

[jodevsa]

Oh, I mean create namespace and a service in that namespace with an explicitly set IP address.

So probably phase 1 will cover your case (I know it's not convenient but I hope this will unblock you):

apiVersion: vpn.example.com/v1alpha1
kind: WireguardPeer
metadata:
name: peer1
spec:
wireguardRef: "my-cool-vpn"
networkPolicy:
egress:
- action: Allow
destination:
IP: 8.8.8.8

[UrfinJuice11]

Hi,

Thanks for the output.

This bug is now fixed and I also added a test for that. Please install the main branch again: https://raw.githubusercontent.com/jodevsa/wireguard-operator/main/release.yaml

Hi, yes. The solution is working well!

1. Creating a peer with a predefined allowed IP address.
2. Creating a pod for this peer with this address in clusterIP
3. The peer can see only this IP.

Thank you!

[jodevsa]

Hi @UrfinJuice11,

I'll write more info soon but will leave a quick hint now:

You'll need to create 2 Wireguard server instances. One called "A" on namespace "A" and another called "B" on namespace "B". if you want a user to only connect to namespace A then create a peer for them on wireguard server A and then create a network policy that makes sure wireguard A only can speak with pods in namespace A

[UrfinJuice11]

I am afraid this is not a solution for my case, since I am going to have tens or hundereds of clients.
But thanks for your reply.

[jodevsa]

are you available for a quick chat about it?

[jodevsa]

can you join:
https://meet.google.com/bii-vnud-mhp?authuser=0&pli=1
?

[UrfinJuice11]

What about a solution, where access controls are implemented not at the CNI level, but at the WG server level?
For example, we create a wireguardpeer instance with smth like allowed_destination_ip definition in yaml (don't mess with allowedIPs in WG config). Then while creating a record in server's wg0.conf this allowed_destination_ip value will be written in PreUP declaration with an iptables rule.
Since services in different namespaces have different IP-addresses, we can use these addresses to differentiate clients' flows.

So, can the feature request be to have custom definitions of PreUp, PostUp and etc. in wg0.conf in wireguard instance yaml. (I tried to add them to the appropriate secret, but the secret was not rewritten)

Also, a question. Is there any means of manual changing config file for WG server?

[jodevsa]

sounds good! so your only interested in allowing each peer to access a couple of k8s services?

if you need the WireguardPeer to be able to access a whole namespace then you somehow need to collect all the IPs in namespace, which continuously changes. pod IPs are not namespace identifiable (there is no way of knowing which namespace this pod IP belongs to). I'm still trying to figure out how the Calico CNI does iptable rules based on namespaces.. I suspect that these rules only apply to the worker of the source or the destination pod but not both as this is the only way to implement such control without having to keep a list of all ips in a namespace.

The idea of keeping a list of all IPs in a namespace is not that crazy and we could bake that inside wireguard-operator. k8s already something similar; If you look at how k8s services are implemented you will see that they rely on storing a list of IPs that matches tha service label. the list of IPs is stored in the endpoints resource can be examined using kubectl get endpoints

[UrfinJuice11]

My situation is the following:

Even now my project can be alive, if there will be a way to mention PostUp iptables rules while creating wireguardpeers. But at this setup I need to:

1. create the wireguardpeer with some IPs (or maybe IP range, like 10.0.0.0/30) in PostUp rules.
2. only then create the Alice namespace and explicitly create her services on the mentioned IP or IP ranges.

This variant is good for me because the Alice's pod will be alive for a short period of time. And all I need is a way to add PostUp and PostDown rules to [Peer] sections of wireguardpeer. Is it possible to add this functionality?

But, I suppose that this solution will cover only my scenario, and in order to make wireguard-operator more universal, the trick with endpoints and DNS-names should be more reasonable.

[jodevsa]

Thanks for the explanation and the diagram.

This variant is good for me because the Alice's pod will be alive for a short period of time. And all I need is a way to add PostUp and PostDown rules to [Peer] sections of wireguardpeer. Is it possible to add this functionality?

Can you give me an example of what iptable command you will be running in PostUp and PostDown of the peer section?

[jodevsa]

I think you'll need to use the peer IP in the iptable rule so we probably need to expose it as an enviroment variable so that your postUp/postDown script can use it

[jodevsa]

I'm thinking of something like this; what do you think?

apiVersion: vpn.example.com/v1alpha1
kind: WireguardPeer
metadata:
  name: peer1
spec:
  wireguardRef: "my-cool-vpn"
  postUp: "iptables -I OUTPUT --source $PEER_IP -d  8.8.8.8 -j DROP"

$PEER_IP is an enviroment variable that will be populated with the actual peer IP

[jodevsa]

another idea that involves some abstractions similar to the ones in NetworkPolicy

apiVersion: vpn.example.com/v1alpha1
kind: WireguardPeer
metadata:
  name: peer1
spec:
  wireguardRef: "my-cool-vpn"
  networkPolicy:
    egress:
     - action: Allow 
        destination:
            IP: 8.8.8.8

I like this more as it would be easier for the average user and doesn't need expertise in iptables. We also need to implement an implicit DENY all behaviour similar to the behaviour of GlobalNetworkPolicy/NetworkPolicy.

Does implementing something like this cover your case?

[UrfinJuice11]

Yes, that's exactly I am talking about.

So, the wg0.conf of the server will be smth. like:

PrivateKey = <server-private-key>
Address = 10.0.0.1/24
ListenPort = <server-port>

[Peer]
PublicKey = <peer1-public-key>
AllowedIPs = 10.0.0.2/32, 192.168.0.2/32
PostUp = iptables -A FORWARD -i wg0 -o eth0 -s 10.0.0.2/32 -d 192.168.0.2/32 -j ACCEPT

[Peer]
PublicKey = <peer2-public-key>
AllowedIPs = 10.0.0.3/32, 192.168.0.3/32
PostUp = iptables -A FORWARD -i wg0 -o eth0 -s 10.0.0.3/32 -d 192.168.0.3/32  -j ACCEPT

But also there should be the contraversal operation like PostDown in order to delete the rules after the peer is dead.

Now, I should think by myself what would be the workflow in my case. I mean if the peer config is created before namespace, than there are no problems, because I can get the destination address 192.168.0.2/32 or 192.168.0.2/30 with kubectl get wireguardpeer and create the namespace with that address.

Anyway it is my own problem. But. I think that adding the PreUp, PostUp, PreDown, PostDown, Address, AllowedIPs sections to the wireguardpeer yaml would be good in any practical case.

[UrfinJuice11]

another idea that involves some abstractions similar to the ones in NetworkPolicy

apiVersion: vpn.example.com/v1alpha1
kind: WireguardPeer
metadata:
  name: peer1
spec:
  wireguardRef: "my-cool-vpn"
  networkPolicy:
    egress:
     - action: Allow 
        destination:
            IP: 8.8.8.8

I like this more as it would be easier for the average user and doesn't need expertise in iptables. We also need to implement an implicit DENY all behaviour similar to the behaviour of GlobalNetworkPolicy/NetworkPolicy.

Does implementing something like this cover your case?

Suppose, that this variant is even better. But it depends on my own flow. Which resources are created first. If there is a chance to use not IP: 8.8.8.8, but namespace: Alice, then it will completely resolve my issue.

[jodevsa]

and create the namespace with that address.

What do you mean by that?

[UrfinJuice11]

Oh, I mean create namespace and a service in that namespace with an explicitly set IP address.

[jodevsa]

phase 1 implementation ideas:

We'll probably rely on iptables-restore to apply new rules every time a Peer is updated. The logic should be idempotent and does not cause any downtime while updating the iptables rules. The operator should inject the iptables rules into the pod of the wireguard server that would look like something like this:

*filter

# rules for peer 10.8.0.2
:10.8.0.2 - [0:0]
-A FORWARD -s 10.8.0.2/32 -j 10.8.0.2
-A 10.8.0.2 -d 10.8.0.1/32 -j ACCEPT
-A 10.8.0.2 -d 100.64.0.10/32 -j ACCEPT
-A 10.8.0.2 -j REJECT --reject-with icmp-port-unreachable

# rules for peer 10.8.0.3
:10.8.0.3 - [0:0]
-A FORWARD -s 10.8.0.3/32 -j 10.8.0.3
-A 10.8.0.3 -d 10.8.0.1/32 -j ACCEPT
-A 10.8.0.3 -d 100.64.0.10/32 -j ACCEPT -m comment --comment "include the json of this rule for debugging purposes"
-A 10.8.0.3 -j REJECT --reject-with icmp-port-unreachable -m comment --comment "implicit reject all"

COMMIT

*nat
:PREROUTING ACCEPT [4234:819045]
:INPUT ACCEPT [14:2125]
:OUTPUT ACCEPT [345:23125]
:POSTROUTING ACCEPT [347:23299]
-A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE
COMMIT

iptables-restore seems to be the only way to update the rules in an atomic fashion

[jodevsa]

more insights about iptables and how to update rules in an atomic way: https://isawan.net/posts/atomic-iptables-script

[jodevsa]

Hi @UrfinJuice11,

I have good news; I'm done with phase 1 #28; it still needs some cleaning but the feature works!! hopefully I'll merge it in the next days

[jodevsa]

here is how the config looks like

spec:
  PrivateKeyRef:
    secretKeyRef:
      key: privateKey
      name: peer20-peer
  address: 10.8.0.2
  egressNetworkPolicies:
  - action: Accept
    protocol: UDP
    to:
      ip: 8.8.8.8
      port: "53"
  - action: Accept
    protocol: TCP
    to:
      ip: 202.78.174.252
      port: "6443"

[jodevsa]

feature is merged; I didn't officially release it yet but you can try it by installing this https://raw.githubusercontent.com/jodevsa/wireguard-operator/main/release.yaml

let me know what do you think

[UrfinJuice11]

Thanks a lot! I'll check it this weekend.

Now can not run the wg-deployment

[jodevsa]

What do you mean?
Can you send me the output of

kubectl get secret wapt-wg --template={{.data.iptable}} | base64 -d

[jodevsa]

btw, why are you deploying wireguard on the wireguard-system namespace? This namespace is supposed to only have the wireguard-operator components and not the actual wireguard server

[jodevsa]

I think wireguard-system.svc.cluster.local does not exist.. you probably need something like wapt-wg.wireguard-system.svc.cluster.local. Again, I do not recommend to use wireguard-system namespace.

[UrfinJuice11]

I think wireguard-system.svc.cluster.local does not exist.. you probably need something like wapt-wg.wireguard-system.svc.cluster.local. Again, I do not recommend to use wireguard-system namespace.

Now created wg server in separate namespace: wapt-wg

The error remains

*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE
COMMIT

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
# start of rules for peer 10.8.0.2
:10-8-0-2 - [0:0]
-A FORWARD -s 10.8.0.2 -j 10-8-0-2
-A 10-8-0-2 -d 172.20.0.5 -p icmp -j ACCEPT
-A 10-8-0-2 -d 10.8.0.2 -j ACCEPT
-A 10-8-0-2 -d 10.96.0.10, wapt-wg.svc.cluster.local -p UDP --dport 53 -j ACCEPT
-A 10-8-0-2 -d 10.13.37.1 -p TCP --dport 9090 -j ACCEPT
-A 10-8-0-2 -j REJECT --reject-with icmp-port-unreachable
# end of rules for peer 10.8.0.2
COMMIT

[jodevsa]

Hi,

Thanks for the output.

This bug is now fixed and I also added a test for that.
Please install the main branch again: https://raw.githubusercontent.com/jodevsa/wireguard-operator/main/release.yaml

[jodevsa]

great to hear that. let me know if you need anything else :)

[jodevsa]

i'll close this issue since you seem to be unblocked now