Rules in the wrong order

[squat]

Rules in the wrong order

[SerialVelocity]

Sure. If the packet doesn't match the first three rules (locally-destined packets), it then hits the catch-all -j MASQUERADE rule and returns from the chain at that point, right?

Here's an output with the amount of packets that have hit the rules:

$ sudo iptables -t nat -nvL KILO-NAT
Chain KILO-NAT (2 references)
 pkts bytes target     prot opt in     out     source               destination
 1743  105K RETURN     all  --  *      *       0.0.0.0/0            172.30.16.0/22       /* Kilo: do not NAT packets destined for the local Pod subnet */
    0     0 RETURN     all  --  *      *       0.0.0.0/0            172.28.128.0/24      /* Kilo: do not NAT packets destined for the Kilo subnet */
    0     0 RETURN     all  --  *      *       172.30.16.0/22       172.28.129.1         /* Kilo: do not NAT packets from local pod subnet to peers */
    0     0 RETURN     all  --  *      *       172.30.16.0/22       192.168.1.0/24       /* Kilo: do not NAT packets from local pod subnet to peers */
 2823  232K MASQUERADE  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Kilo: NAT remaining packets */
    0     0 RETURN     all  --  *      *       172.30.16.0/22       172.30.4.0/22        /* Kilo: do not NAT packets from local pod subnet to remote pod subnets */
    0     0 RETURN     all  --  *      *       172.30.16.0/22       172.30.0.0/22        /* Kilo: do not NAT packets from local pod subnet to remote pod subnets */
    0     0 RETURN     all  --  *      *       172.30.16.0/22       172.30.12.0/22       /* Kilo: do not NAT packets from local pod subnet to remote pod subnets */
    0     0 RETURN     all  --  *      *       172.30.16.0/22       172.30.8.0/22        /* Kilo: do not NAT packets from local pod subnet to remote pod subnets */
    0     0 RETURN     all  --  *      *       0.0.0.0/0            10.255.255.253       /* Kilo: do not NAT packets destined for the local private IP */

Here's an http call from a source host:

root@test-pod:/# ifconfig eth0
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.30.16.3  netmask 255.255.252.0  broadcast 0.0.0.0
        ether 5e:59:42:62:aa:f5  txqueuelen 0  (Ethernet)
        RX packets 14861990  bytes 27847081553 (27.8 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 23475078  bytes 48491026535 (48.4 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
root@test-pod:/# curl 172.30.8.4
curl: (7) Failed to connect to 172.30.8.4 port 80: Connection refused

Here's a tcpdump from the destination host:

$ sudo tcpdump -vvvni kilo0 "host 172.30.8.4"
dropped privs to tcpdump
tcpdump: listening on kilo0, link-type RAW (Raw IP), capture size 262144 bytes
17:20:39.879485 IP (tos 0x0, ttl 63, id 56731, offset 0, flags [DF], proto TCP (6), length 60)
    172.28.128.1.43858 > 172.30.8.4.80: Flags [S], cksum 0xd30b (correct), seq 719155311, win 29200, options [mss 1460,sackOK,TS val 362595862 ecr 0,nop,wscale 9], length 0
17:20:39.879567 IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 40)
    172.30.8.4.80 > 172.28.128.1.43858: Flags [R.], cksum 0x88a0 (correct), seq 0, ack 719155312, win 0, length 0
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel

The packet has been masqueraded and looks like it is coming from the host when really it should be coming from the pod IP (since it is heading to another pod IP).

From looking at:

kilo/pkg/iptables/iptables.go

Lines 275 to 298 in c93fa1e

	// MasqueradeRules returns a set of iptables rules that are necessary
	// to NAT traffic from the local Pod subnet to the Internet and out of the Kilo interface.
	func MasqueradeRules(kilo, private, localPodSubnet *net.IPNet, remotePodSubnet, peers []*net.IPNet) []Rule {
	var rules []Rule
	rules = append(rules, &chain{"nat", "KILO-NAT", nil})
	// NAT packets from Kilo interface.
	rules = append(rules, &rule{"mangle", "PREROUTING", []string{"-m", "comment", "--comment", "Kilo: jump to mark chain", "-i", "kilo+", "-j", "MARK", "--set-xmark", "0x1107/0x1107"}, nil})
	rules = append(rules, &rule{"nat", "POSTROUTING", []string{"-m", "comment", "--comment", "Kilo: NAT packets from Kilo interface", "-m", "mark", "--mark", "0x1107/0x1107", "-j", "KILO-NAT"}, nil})
	// NAT packets from pod subnet.
	rules = append(rules, &rule{"nat", "POSTROUTING", []string{"-m", "comment", "--comment", "Kilo: jump to NAT chain", "-s", localPodSubnet.String(), "-j", "KILO-NAT"}, nil})
	rules = append(rules, &rule{"nat", "KILO-NAT", []string{"-m", "comment", "--comment", "Kilo: do not NAT packets destined for the local Pod subnet", "-d", localPodSubnet.String(), "-j", "RETURN"}, nil})
	rules = append(rules, &rule{"nat", "KILO-NAT", []string{"-m", "comment", "--comment", "Kilo: do not NAT packets destined for the Kilo subnet", "-d", kilo.String(), "-j", "RETURN"}, nil})
	rules = append(rules, &rule{"nat", "KILO-NAT", []string{"-m", "comment", "--comment", "Kilo: do not NAT packets destined for the local private IP", "-d", private.String(), "-j", "RETURN"}, nil})
	for _, r := range remotePodSubnet {
	rules = append(rules, &rule{"nat", "KILO-NAT", []string{"-m", "comment", "--comment", "Kilo: do not NAT packets from local pod subnet to remote pod subnets", "-s", localPodSubnet.String(), "-d", r.String(), "-j", "RETURN"}, nil})
	}
	for _, p := range peers {
	rules = append(rules, &rule{"nat", "KILO-NAT", []string{"-m", "comment", "--comment", "Kilo: do not NAT packets from local pod subnet to peers", "-s", localPodSubnet.String(), "-d", p.String(), "-j", "RETURN"}, nil})
	}
	rules = append(rules, &rule{"nat", "KILO-NAT", []string{"-m", "comment", "--comment", "Kilo: NAT remaining packets", "-j", "MASQUERADE"}, nil})
	return rules
	}

It looks like you put the rules in the right order in code but you probably don't insert the rules correctly if some of them already exist in iptables.

[squat]

Hey, thanks for taking a look. Can you elaborate? It looks to me like they do get hit and provide functionality. In fact, turning them off breaks networking.

[squat]

@SerialVelocity thanks once again for the super detailed write up as well as for digging through the code base. I think your analysis is exactly right: order matters and the code doesn't check to make sure that order is preserved when the rules change.