IPIP rules in security groups not applying properly

[vexxhost]

IPIP rules in security groups not applying properly

[mnaser]

I just filed an issue into CAPI and then we can track that up to Neutron if that's the path to go, but a revert would be nice for now.

[mnaser]

I just did some more troubleshooting. setting this Security Group openstack security group rule create 76192aca-20eb-43e6-bd1c-331979e57157 --remote-group ba33108a-ceb1-4ef0-98cd-455fa916166f --protocol ipip --ingress creates this iptables rule: 0 0 RETURN 94 -- * * 0.0.0.0/0 0.0.0.0/0 match-set NIPv476192aca-20eb-43e6-bd1c- src So that does get translated into iptables 'correctly' as ipip = proto 94 according to https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml
So it seems that the protocol that needs to be allowed on iptables would be '4': 'ipencap' instead of '94': 'ipip'...? https://github.com/openstack/neutron-lib/blob/75ee705ed5f4144483686d7e237ba78afe6d1dff/neutron_lib/constants.py#L459

This is some good research, I wonder if inside iptables, it used to use 94 at some point and so the translation is failing depending on the OS.

yeap, 94 is KA9Q NOS compatible IP over IP tunneling and here is a bit more explanation https://puck.nether.net/lists/juniper-nsp/1269.html

useful to know!

anyways, we've got the fix merged and backported into 0.7, the CAPO team will make a release on Monday for 0.7.1 and by using that these issues will be solved :)

kubernetes-sigs/cluster-api-provider-openstack#1493

[robincron]

looks like this is a default from here and we could override it if we wanted
https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/5a17c37f0a0d4d44211f2339a6893752b43e11da/pkg/cloud/services/networking/securitygroups_rules.go#L79-L115

[mnaser]

We should probably bubble this up to the Cluster API project, which might bubble up to Neutron, since this feels like a Neutron bug.

[fnpanic]

@mnaser I think this is the right approach fixing this upstream. However how can we mitigate this in fast way in the meantime?

[robincron]

I just did some more troubleshooting.
setting this Security Group
openstack security group rule create 76192aca-20eb-43e6-bd1c-331979e57157 --remote-group ba33108a-ceb1-4ef0-98cd-455fa916166f --protocol ipip --ingress
creates this iptables rule:
0 0 RETURN 94 -- * * 0.0.0.0/0 0.0.0.0/0 match-set NIPv476192aca-20eb-43e6-bd1c- src
So that does get translated into iptables 'correctly' as ipip = proto 94 according to https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml

So it seems that the protocol that needs to be allowed on iptables would be '4': 'ipencap' instead of '94': 'ipip'...?
https://github.com/openstack/neutron-lib/blob/75ee705ed5f4144483686d7e237ba78afe6d1dff/neutron_lib/constants.py#L459

[fnpanic]

According to IANA and the Calico Documentaiton Calico uses IP in IP which is IP protocol number 4 not 94. https://docs.tigera.io/calico/3.25/networking/configuring/vxlan-ipip

Cluster API seems to set this wrong from my point of view.

EDIT: @mnaser

This is causing the problem. It is only merged in main but not 0.7 release. What concern me that is passed CI flawless and makes the cluster unusable.

replace "4" with "ipip"
kubernetes-sigs/cluster-api-provider-openstack#1290

but still it is a neutron bug
https://docs.openstack.org/api-ref/network/v2/index.html?expanded=create-security-group-rule-detail#security-group-rules-security-group-rules

[mnaser]

I just did some more troubleshooting. setting this Security Group openstack security group rule create 76192aca-20eb-43e6-bd1c-331979e57157 --remote-group ba33108a-ceb1-4ef0-98cd-455fa916166f --protocol ipip --ingress creates this iptables rule: 0 0 RETURN 94 -- * * 0.0.0.0/0 0.0.0.0/0 match-set NIPv476192aca-20eb-43e6-bd1c- src So that does get translated into iptables 'correctly' as ipip = proto 94 according to https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml

So it seems that the protocol that needs to be allowed on iptables would be '4': 'ipencap' instead of '94': 'ipip'...? https://github.com/openstack/neutron-lib/blob/75ee705ed5f4144483686d7e237ba78afe6d1dff/neutron_lib/constants.py#L459

This is some good research, I wonder if inside iptables, it used to use 94 at some point and so the translation is failing depending on the OS.

[mnaser]

According to IANA and the Calico Documentaiton Calico uses IP in IP which is IP protocol number 4 not 94. https://docs.tigera.io/calico/3.25/networking/configuring/vxlan-ipip

Cluster API seems to set this wrong from my point of view.

EDIT: @mnaser

This is causing the problem. It is only merged in main but not 0.7 release. What concern me that is passed CI flawless and makes the cluster unusable.

replace "4" with "ipip" kubernetes-sigs/cluster-api-provider-openstack#1290

but still it is a neutron bug https://docs.openstack.org/api-ref/network/v2/index.html?expanded=create-security-group-rule-detail#security-group-rules-security-group-rules

For the record, we're not running into that issue.. however, that mapping is not the source of truth, see this: https://github.com/openstack/neutron/blob/master/neutron/agent/linux/iptables_firewall.py#L745-L773

I'm curious to see if your environments somehow are seeing a mismatch... could you double check that?

[mnaser]

ok, have a fix up:

kubernetes-sigs/cluster-api-provider-openstack#1489

now this is technically more of a CAPI issue than the driver, but we'll close this once we fix this upstream and also fix it inside Atmosphere

[mnaser]

two things:

• fix for hack/stack.sh is done, so this won't happen anymore
• fix for atmosphere is done: vexxhost/atmosphere#334

will keep this open to track CAPO :)

[mnaser]

ugh, not so quick

https://github.com/kubernetes-sigs/cluster-api-provider-openstack/blob/v0.7.0/pkg/cloud/services/networking/securitygroups_rules.go#L163-L176

this was included in 0.7.0, going to keep working with upstream on a fix soon, if not, we'll do our own local patch for it.

[okozachenko1203]

I just did some more troubleshooting. setting this Security Group openstack security group rule create 76192aca-20eb-43e6-bd1c-331979e57157 --remote-group ba33108a-ceb1-4ef0-98cd-455fa916166f --protocol ipip --ingress creates this iptables rule: 0 0 RETURN 94 -- * * 0.0.0.0/0 0.0.0.0/0 match-set NIPv476192aca-20eb-43e6-bd1c- src So that does get translated into iptables 'correctly' as ipip = proto 94 according to https://www.iana.org/assignments/protocol-numbers/protocol-numbers.xhtml
So it seems that the protocol that needs to be allowed on iptables would be '4': 'ipencap' instead of '94': 'ipip'...? https://github.com/openstack/neutron-lib/blob/75ee705ed5f4144483686d7e237ba78afe6d1dff/neutron_lib/constants.py#L459

This is some good research, I wonder if inside iptables, it used to use 94 at some point and so the translation is failing depending on the OS.

yeap, 94 is KA9Q NOS compatible IP over IP tunneling and here is a bit more explanation https://puck.nether.net/lists/juniper-nsp/1269.html