cilium-cni touches MTU of default interface

[learnitall]

cilium-cni touches MTU of default interface

[learnitall]

Last week I did some testing to confirm that this issue stems from retrieveIfIndexFromMAC. The cilium-cni plugin binary uses netlink.LinkSetMTU to change interface MTUs. I modified the code for this function to output the following:

1. The name of the executable currently running (ie os.Executable()).
2. The attributes of the Link object passed to LinkSetMTU (ie link.Attrs()).
3. The current time.
4. The MTU passed to LinkSetMTU.
5. The runtime stack (ie debug.Stack()).

I was able to gather data to confirm the cilium-cni is modifying default interface MTUs inside of retrieveIfIndexFromMAC after receiving a CNI Add request. For example when running with WireGuard on an EKS cluster with ENI IPAM, interfaces on the node such as eth0 and/or eth1 will have their MTU lowered to 8192. (Note, this is running with a version of Cilium that does not include #33304.) The bug can be triggered by scheduling a bunch of Pods.

Here's roughly what the flow of the relevant code path looks like:

I chatted with @clay-moss who was able to make sense of this. The cilium-cni receives a CNI Add request, then reaches out to the Cilium Agent to request IPAM information. When running in Azure, AWS ENI, or AlibabaCloud IPAM mode, the Cilium Agent may respond with an IP that is associated with one of the default interfaces of a node. This is expected behavior and documented (for example, see here for AWS ENI). When the MAC address of one of the default interfaces on the node is passed to retrieveIfIndexFromMAC, a call to netlink.LinkSetMTU is made which changes the MTU of the default interface.

This code path has been in place since fac1802, which was committed back in 2020. My assumption is that we haven't caught this issue because normally the cilium-cni just sets the MTU on a default interface to the MTU value that the Cilium Agent read from the interface during startup. A user may come across this issue if they try to manually change the MTU of a default interface while Cilium is running, as eventually the interface's MTU will be restored.

Out of all the engineers I've discussed this with, the underlying assumption is that Cilium should not change the MTU of existing interfaces, especially those that are not managed by Cilium on a node. My recommendation is for us to remove the MTU management from retrieveIfIndexFromMAC, which will restore this assumption. Additionally, we should write an integration test which checks to see if this assumption is being held true.

[learnitall]

In the previous comment, I mentioned that retrieveIfIndexFromMAC was added in commit fac1802, but that was incorrect. It was actually added in commit 26308b6, which was part of #11073.

[learnitall]

Actually, @joestringer found that the code was added in commit 27ff371, which was part of #8347 that added ENI support to Cilium