VPC CNI plugins for Amazon ECS and EKS.
License: Apache License 2.0
VPC CNI plugins for Amazon ECS and Amazon EKS.
If you think you’ve found a potential security issue, please do not post it in the Issues. Instead, please follow the instructions here or email AWS security directly.
This library is licensed under the Apache 2.0 License.
When I enable a policy to allow pods in one namespace to communicate with services in the same namespace windows pods can no longer communicate with any service inside that namespace. The rest of the cluster (linux nodes and pods) work as expected and can communicate.
After a lot of trial and error I could pin point the issue on the fact, that the traffic from the pods on the windows node is not routed directly but NATed through the host. (I documented a lot of the process in this issue I created on the calico project projectcalico/calico#4936)
As a workaround I whitelisted the IP of the windows host in one of the namespaces which is very ugly and not really scalable.
As a start it would be nice to know how you guys intended this to function since I can't really find any useful documentation specifying what the actual expected behaviour is.
For running windows nodes in EKS, I expect kube-proxy to route the traffic directly to the VPC as is the case for the rest of the cluster running on linux nodes.
Kube-proxy on windows applies NAT to the traffic emerging from the pods.
EKS with k8s 1.19
Calico version 3.20
WindowsServer2019FullContainer AMI for the windows nodes
Linux nodes are managed
Since v1.69.0 of the ECS agent the plugin crashes, with the latest version, 1.77.0, the service can start a task, but it cannot connect to the network. As per AWS recommendations we're using awsvpc mode and blocking the IMDS endpoint, which appears to be the cause of the issue.
We see an error in the logs, see below, when the plugin attempts to add the blackhole route for the IPv6 address of the IMDS, this line in the code appears to be what's failing.
The AMI that we use for the EC2 instance does not have IPv6 enabled for security reasons.
The plugin handles the resulting error gracefully, or provides some means of disabling the IPv6 support to avoid the error.
The plugin crashes and fails to finish configuring the network routes for the task, e.g. the default route is missing from the task route table, etc.
Route table with latest agent and ECS_AWSVPC_BLOCK_IMDS=true:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
169.254.169.254 * 255.255.255.255 UH 0 0 0 *
169.254.170.2 169.254.172.1 255.255.255.255 UGH 0 0 0 ecs-eth0
169.254.172.1 169.254.172.1 255.255.255.255 UGH 0 0 0 ecs-eth0
Route table with latest agent and ECS_AWSVPC_BLOCK_IMDS=false:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default ip-10-116-128-1 0.0.0.0 UG 0 0 0 eth0
10.116.128.128 * 255.255.255.224 U 0 0 0 eth0
169.254.170.2 169.254.172.1 255.255.255.255 UGH 0 0 0 ecs-eth0
169.254.172.1 169.254.172.1 255.255.255.255 UGH 0 0 0 ecs-eth0
Route table with v1.68.2 of the agent and ECS_AWSVPC_BLOCK_IMDS=true:
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default ip-10-116-128-1 0.0.0.0 UG 0 0 0 eth0
10.116.128.160 * 255.255.255.224 U 0 0 0 eth0
169.254.169.254 * 255.255.255.255 UH 0 0 0 *
169.254.170.2 169.254.172.1 255.255.255.255 UGH 0 0 0 ecs-eth0
169.254.172.1 169.254.172.1 255.255.255.255 UGH 0 0 0 ecs-eth0
docker info:
Client:
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc., v0.0.0+unknown)
Server:
Containers: 3
Running: 3
Paused: 0
Stopped: 0
Images: 3
Server Version: 20.10.25
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 1e1ea6e986c6c86565bc33d52e34b81b3e2bc71f
runc version: f19387a6bec4944c770f7668ab51c4348d9c2f38
init version: de40ad0
Security Options:
seccomp
Profile: default
Kernel Version: 5.10.197-186.748.amzn2.x86_64
Operating System: Amazon Linux 2
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.43GiB
Name: ip-X-X-X-X.eu-west-2.compute.internal
ID: U742:Y4OW:URO4:EH6C:LBN5:5HMC:HHOP:LWGQ:6RIV:FTJY:KJW3:QBBQ
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
EKS node upgrade failing when using istio CNI plugin,
EKS is working fine when I am installing aws CNI then istio with istio CNI, both are running fine with eks and if eks autoscale itself new node up successfully,
But at later stage when i trying to upgrade nodegroup the entire upgrade progress is failing, AWS CNI pod (aws-node) is failing on new upgrade node due to which node upgrade process is failing.
Apologies if this isn't the correct place to ask such a question.
Given an ECS container instance with an AppMesh-enabled task (an application container and an Envoy proxy sidecar), how can I view the iptables rules created in https://github.com/aws/amazon-vpc-cni-plugins/blob/master/plugins/aws-appmesh/plugin/commands.go? I'm looking to verify these rules using the iptables command.
I don't see them on the host, or in any of the Docker network namespaces for the application or Envoy containers. Any tips would be much appreciated!
As a first step for migrating from godep to go modules, we need to setup a go module.
When containers are provisioned through containerd on Windows (instead of docker), the CNI plugin fails
on the ADD operation.
When containers are started through containerd on Windows, the ADD operation fails due to this logic here -> https://github.com/aws/amazon-vpc-cni-plugins/blob/master/plugins/vpc-shared-eni/network/bridge_windows.go#L361
It seems like a simple fix would be to just assume that the block that has the error is invoked in the case of containerd/cri usage, in which case we can probably assume that it's only being invoked on the infra container.
The CNI should ADD successfully for containers started through containerd.
The CNI fails the ADD operation for containers started through containerd.
2021-08-26T19:03:55Z [INFO] Executing ADD with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Names
ervers:[172.20.0.10] Domain: Search:[addon-manager-logging.svc.cluster.local svc.cluster.local cluster.local] Options:[]}} ENIName: ENIMACAddress:0a:df:16:54
:db:51 ENIIPAddress:10.17.42.208/24 VPCCIDRs:[{IP:10.17.32.0 Mask:ffffe000} {IP:10.17.64.0 Mask:ffffe000} {IP:10.17.96.0 Mask:ffffe000}] BridgeType:L3 Bridge
NetNSPath: IPAddress:10.17.42.99/24 GatewayIPAddress:10.17.42.1 InterfaceType:veth TapUserID:0 Kubernetes:{Namespace:addon-manager-logging PodName:fluent-bit
-windows-tb4ds PodInfraContainerID:20fe247f82deb6d6311c0318918999fa40d4b715b61cc97f3c264e6bb1944f79 ServiceCIDR:172.20.0.0/16}} ContainerID:20fe247f82deb6d63
11c0318918999fa40d4b715b61cc97f3c264e6bb1944f79 Netns:504314d1-16b7-47e3-8cc2-b80c1510d1b5 IfName:eth0 Args:K8S_POD_INFRA_CONTAINER_ID=20fe247f82deb6d6311c03
18918999fa40d4b715b61cc97f3c264e6bb1944f79;K8S_POD_UID=fa8bdee8-3c26-4657-8b7e-b76a1fda5601;IgnoreUnknown=1;K8S_POD_NAMESPACE=addon-manager-logging;K8S_POD_N
AME=fluent-bit-windows-tb4ds.
2021-08-26T19:03:55Z [INFO] Running on HNS version: {Major:9 Minor:5}
2021-08-26T19:03:55Z [INFO] Found existing HNS network vpcbr0adf1654db51.
2021-08-26T19:03:55Z [ERROR] Failed to parse netns 504314d1-16b7-47e3-8cc2-b80c1510d1b5 of container 20fe247f82deb6d6311c0318918999fa40d4b715b61cc97f3c264e6b
b1944f79
Seeing a couple of leaked endpoints (possibly) when I am scaling the deployment and before the pod goes to Running state, I am scaling down the deployment.
Add Request for pod "win-webserver-75bc4c4c6f-chxlz" with container ID "9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b"
563 2020-08-06T01:34:12Z [INFO] Executing ADD with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[10.100.0.10] Domain: Search:[default.svc.cluster.local svc.cluster.local cluster.local] Options:[]}} ENIName: ENIMACAddress:02:cb:e2:d2:e5:e7 ENIIPAddress:192.168.1.161/19 VPCCIDRs:[{IP:192.168 .0.0 Mask:ffff0000}] BridgeType:L3 BridgeNetNSPath: IPAddress:192.168.28.156/19 GatewayIPAddress:192.168.0.1 InterfaceType:veth TapUserID:0 Kubernetes:{Namespace:default PodName:win-webserver-75bc4c4c6f-chxlz PodInfraContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b ServiceCIDR:10.100.0.0/16}} ContainerID:9b1eb5e026f3e03959fd1263 6265d8ba19718453d8b1642e5a014874b3e47d1b Netns:none IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=win-webserver-75bc4c4c6f-chxlz;K8S_POD_INFRA_CONTAINER_ID=9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
Created HNS Endpoint for the Pod
540 2020-08-06T01:34:11Z [INFO] Creating HNS endpoint: {"Name":"cid-9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b","VirtualNetworkName":"vpcbr02cbe2d2e5e7","Policies":[{"Type":"OutBoundNAT","ExceptionList":["192.168.0.0/16","10.100.0.0/16"]},{"Type":"ROUTE","DestinationPrefix":"10.100.0.0/16","NeedEncap":true},{"Type":"ROUTE","DestinationPre fix":"192.168.1.161/32","NeedEncap":true}],"IPAddress":"192.168.28.156","DNSSuffix":"default.svc.cluster.local,svc.cluster.local,cluster.local","DNSServerList":"10.100.0.10","PrefixLength":19}
541 2020-08-06T01:34:11Z [INFO] Received HNS endpoint response: &{Id:EFDCCA66-44F3-4F61-BC64-C8DD620DD46E Name:cid-9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b VirtualNetwork:AEC44415-A934-4370-A616-3F8E0581FB2E VirtualNetworkName:vpcbr02cbe2d2e5e7 Policies:[[123 34 69 120 99 101 112 116 105 111 110 76 105 115 116 34 58 91 34 49 57 50 46 49 54 56 46 48 46 48 47 49 54 34 44 34 49 48 46 49 48 48 46 48 46 48 47 49 54 34 93 44 34 84 121 112 101 34 58 34 79 117 116 66 111 117 110 100 78 65 84 34 125] [123 34 68 101 115 116 105 110 97 116 105 111 110 80 114 101 102 105 120 34 58 34 49 48 46 49 48 48 46 48 46 48 47 49 54 34 44 34 78 101 101 100 69 110 99 97 112 34 58 116 114 117 101 44 34 84 121 11 2 101 34 58 34 82 79 85 84 69 34 125] [123 34 68 101 115 116 105 110 97 116 105 111 110 80 114 101 102 105 120 34 58 34 49 57 50 46 49 54 56 46 49 46 49 54 49 47 51 50 34 44 34 78 101 101 100 69 110 99 97 112 34 58 116 114 117 101 44 34 84 121 112 101 34 58 34 82 79 85 84 69 34 125] [123 34 84 121 112 101 34 58 34 76 50 68 114 105 118 101 114 34 125]] MacA ddress:00-15-5D-3F-AD-0B IPAddress:192.168.28.156 DNSSuffix:default.svc.cluster.local,svc.cluster.local,cluster.local DNSServerList:10.100.0.10 GatewayAddress:192.168.0.1 EnableInternalDNS:false DisableICC:false PrefixLength:19 IsRemoteEndpoint:false Namespace:<nil>}.
542 2020-08-06T01:34:11Z [INFO] Attaching HNS endpoint EFDCCA66-44F3-4F61-BC64-C8DD620DD46E to container 9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
Received the ADD request again (twice), we are checking if the HNS Endpoint exist then not to recreate it (which is the expected behavior)
Second ADD request,
2020-08-06T01:34:12Z [INFO] Executing ADD with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[10.100.0.10] Domain: Search:[default.svc.cluster.local svc.cluster.local cluster.local] Options:[]}} ENIName: ENIMACAddress:02:cb:e2:d2:e5:e7 ENIIPAddress:192.168.1.161/19 VPCCIDRs:[{IP:192.168.0.0 Mask:ffff0000}] BridgeType:L3 BridgeNetNSPath: IPAddress:192.168.28.156/19 GatewayIPAddress:192.168.0.1 InterfaceType:veth TapUserID:0 Kubernetes:{Namespace:default PodName:win-webserver-75bc4c4c6f-chxlz PodInfraContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b ServiceCIDR:10.100.0.0/16}} ContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b Netns:none IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=win-webserver-75bc4c4c6f-chxlz;K8S_POD_INFRA_CONTAINER_ID=9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
2020-08-06T01:34:13Z [INFO] Found existing HNS endpoint cid-9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
2020-08-06T01:34:13Z [INFO] HNS endpoint cid-9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b is already attached to container ID 9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
Third ADD request,
2020-08-06T01:34:15Z [INFO] Executing ADD with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[10.100.0.10] Domain: Search:[default.svc.cluster.local svc.cluster.local cluster.local] Options:[]}} ENIName: ENIMACAddress:02:cb:e2:d2:e5:e7 ENIIPAddress:192.168.1.161/19 VPCCIDRs:[{IP:192.168.0.0 Mask:ffff0000}] BridgeType:L3 BridgeNetNSPath: IPAddress:192.168.28.156/19 GatewayIPAddress:192.168.0.1 InterfaceType:veth TapUserID:0 Kubernetes:{Namespace:default PodName:win-webserver-75bc4c4c6f-chxlz PodInfraContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b ServiceCIDR:10.100.0.0/16}} ContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b Netns:none IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=win-webserver-75bc4c4c6f-chxlz;K8S_POD_INFRA_CONTAINER_ID=9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
2020-08-06T01:34:15Z [INFO] Found existing HNS endpoint cid-9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
2020-08-06T01:34:15Z [INFO] HNS endpoint cid-9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b is already attached to container ID 9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
Now the Del Command is invoked as the pod object is deleted,
2020-08-06T01:34:15Z [INFO] Executing DEL with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[10.100.0.10] Domain: Search:[{%namespace%}.svc.cluster.local svc.cluster.local cluster.local] Options:[]}} ENIName: ENIMACAddress:02:cb:e2:d2:e5:e7 ENIIPAddress:192.168.1.161/19 VPCCIDRs:[{IP:192.168.0.0 Mask:ffff0000}] BridgeType:L3 BridgeNetNSPath: IPAddress:<nil> GatewayIPAddress:192.168.0.1 InterfaceType:veth TapUserID:0 Kubernetes:{Namespace:default PodName:win-webserver-75bc4c4c6f-chxlz PodInfraContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b ServiceCIDR:10.100.0.0/16}} ContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b Netns:none IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=win-webserver-75bc4c4c6f-chxlz;K8S_POD_INFRA_CONTAINER_ID=9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
When the plugin is trying to detach the HNS Endpoint from the container we receive an error that the container no longer exists (which is strange considering the DEL command has not succeeded yet)
2020-08-06T01:34:15Z [INFO] Detaching HNS endpoint EFDCCA66-44F3-4F61-BC64-C8DD620DD46E from container 9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b netns.
2020-08-06T01:34:15Z [ERROR] Failed to delete endpoint, ignoring: A virtual machine or container with the specified identifier does not exist.
I am new to CNI Plugin side of the code, but here is my understanding
However, If the above scenario can happen ideally then we should clean up the endpoint before returning the error.
From the code I can see that we don't clean up the HNS Endpoint when we get error related to container no longer exists in the detach workflow.
// Detach the HNS endpoint from the container's network namespace.
log.Infof("Detaching HNS endpoint %s from container %s netns.", hnsEndpoint.Id, ep.ContainerID)
err = hcsshim.HotDetachEndpoint(ep.ContainerID, hnsEndpoint.Id)
if err != nil {
return err
}
Using Build with the changes from Pull Request #46 [Which resolves a totally separate issue]
Hi, I'm reaching to ask if there is a plan to upgrade go version to address the CVE-2023-39325 : GHSA-4374-p667-p6c8.
Thank you.
Upgrade CNI dependency to v1.1.2 and update all CNI plugins to support spec v1.0.0
Amazon ECS is upgrading its CNI components across EC2 and Fargate to spec v1.0.0 so that we can build a single networking stack and build process.
IPAMD is restarting with the below messages.
{"level":"debug","ts":"2020-11-21T12:18:10.565Z","caller":"awsutils/awsutils.go:388","msg":"Update ENI eni-xxxxxxxxxx"}
{"level":"error","ts":"2020-11-21T12:18:10.773Z","caller":"aws-k8s-agent/main.go:28","msg":"Initialization failure: ipamd: can not initialize with AWS SDK interface: refreshSGIDs: unable to update the ENI's SG: InvalidNetworkInterfaceID.NotFound: The networkInterface ID 'eni-xxxxxxxxxxxx' does not exist\n\tstatus code: 400, request id: aaaaaa-bbbbbb-cccccc-ccccc-sssssss"}
From awsutils l458 We understand ModifyNetworkInterfaceAttribute action is triggering the above error message.
From the Cloud trail API calls I was able to confirm that the ENI was created and deleted by VPC cni however. They were not cleared from EC2 metadata. Since metadataMACPath function will use the below snippet to make the metadata call which will gather the eni-id We are seeing the above issue.
eniID, err = cache.ec2Metadata.GetMetadata( + eniMAC + metadataInterface)
I have confirmed by running the below curl call that the deleted ENI is still present on the metadata.
curl 169.254.169.254/2020-10-27/meta-data/network/interfaces/macs/<macaddress>/<interface-id>
IPAMD should be able to handle out-of-synced IMDS in above cases. As This will cause aws-node to be in crashloopback state and cause pods scheduling onto the node to not have any ip's causing them to be stuck in creating state.
similar issue form the past #1177
Why is this needed:
CNI should handle similar issue related to imds gracefully instead of getting the node and all pods stuck.
Hello. We have been seeing an issue intermittently where pods will become stuck in container creating with the following error or similar:
Warning FailedCreatePodSandBox 56m kubelet, ip-10-209-103-40.us-west-2.compute.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d20eb84d0be212c52e734d4c034f9ea29fb3a98c78d4d4cb111cf536f45298ea" network for pod "<redacted>": networkPlugin cni failed to set up pod "<redacted>" network: hnsCall failed in Win32: An address provided is invalid or reserved. (0x803b002f)
K8s version: 1.16
This pod is being scheduled on a Windows Worker Node using the amazon/Windows_Server-1909-English-Core-EKS_Optimized-1.16-2020.06.10 AMI.
We have also observed errors in the CNI logs as follows:
**2020-07-22T19:59:53Z [ERROR] Failed to parse netconfig from args: failed to parse Kubernetes args: failed to get pod settingsapi-test-connection: pods "settingsapi-test-connection" not found.
2020-07-22T19:59:53Z [ERROR] CNI command failed: failed to parse Kubernetes args: failed to get pod settingsapi-test-connection: pods "settingsapi-test-connection" not found
2020-07-22T19:59:53Z [INFO] Plugin vpc-shared-eni version executing CNI command.
2020-07-22T19:59:53Z [ERROR] Failed to parse netconfig from args: failed to parse Kubernetes args: failed to get pod tmp-shelly3: pods "tmp-shelly3" not found.
2020-07-22T19:59:53Z [ERROR] CNI command failed: failed to parse Kubernetes args: failed to get pod tmp-shelly3: pods "tmp-shelly3" not found
2020-07-22T19:59:53Z [INFO] Plugin vpc-shared-eni version executing CNI command.
2020-07-22T19:59:53Z [ERROR] Failed to parse netconfig from args: failed to parse Kubernetes args: failed to get pod <redacted>: pods "<redacted>" not found.
2020-07-22T19:59:53Z [ERROR] CNI command failed: failed to parse Kubernetes args: failed to get pod <redacted>: pods "<redacted>" not found
2020-07-22T19:59:58Z [INFO] Plugin vpc-shared-eni version executing CNI command.
PS C:\ProgramData\Amazon\EKS\logs>
**
These errors appear to be referencing pods that no longer exist (and haven't for some time), are repeated constantly, and do not seem to directly correlate in start time to the incident above. The only way to stop the CNI from logging this second set errors seems to be to terminate the node, they even persist through reboot.
I'm happy to provide any more details that I can if needed.
I'm running EKS K8S 1.30 with EC2 Windows workers nodes only, coredns runing on fargate
There is an issue with dnsConfig that are ignored for Windows pods:
spec:
dnsPolicy: "None"
dnsConfig:
nameservers:
- 192.168.1.100
searches:
- custom.domain
C:\app>ipconfig /all
Windows IP Configuration
Host Name . . . . . . . . . . . . : eks-app
Primary Dns Suffix . . . . . . . : custom.domain
Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : custom.namespace.svc.cluster.local
svc.cluster.local
cluster.local
Ethernet adapter vEthernet (cid-54890e7f-6a27-434b-ab4c-12ba9c74c1e6):
Connection-specific DNS Suffix . : custom.namespace.svc.cluster.locall
Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Container Adapter
Physical Address. . . . . . . . . : 00-15-5D-21-52-E4
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
Link-local IPv6 Address . . . . . : fe80::e67b:4cd8:d426:be86%18(Preferred)
IPv4 Address. . . . . . . . . . . : 10.50.130.207(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.50.130.1
DNS Servers . . . . . . . . . . . : 172.20.0.10
NetBIOS over Tcpip. . . . . . . . : Disabled
Connection-specific DNS Suffix Search List :
custom.namespace.svc.cluster.local
svc.cluster.local
cluster.local
Expected to happen:
Windows IP Configuration
Host Name . . . . . . . . . . . . : eks-app
Primary Dns Suffix . . . . . . . : custom.domain
Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : **custom.domain**
Ethernet adapter vEthernet (cid-54890e7f-6a27-434b-ab4c-12ba9c74c1e6):
Connection-specific DNS Suffix . : **custom.domain**
Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Container Adapter
Physical Address. . . . . . . . . : 00-15-5D-21-52-E4
DHCP Enabled. . . . . . . . . . . : No
Autoconfiguration Enabled . . . . : Yes
Link-local IPv6 Address . . . . . : fe80::e67b:4cd8:d426:be86%18(Preferred)
IPv4 Address. . . . . . . . . . . : 10.50.130.207(Preferred)
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 10.50.130.1
DNS Servers . . . . . . . . . . . : **192.168.1.100**
NetBIOS over Tcpip. . . . . . . . : Disabled
Connection-specific DNS Suffix Search List :
**custom.domain**
shared-vpc-eni plugin enables proxy-arp on the bridge, however, the default proxy_delay(80) is used, where linux will randomly add delays before respond to ARP request. This can cause significant latency especially in DNS requests if the dns
In our environment, we configured pod networking using shared-vpc-eni plugin, and pod have a route rule that will route /19 addresses with ARP resolution through the bridge setup by shared-vpc-eni plugin. The bridge is now responding the ARP request with a random delay up to 800ms.
21:39:35.304307 IP ip-192-168-145-95.us-west-2.compute.internal.54962 > coredns-7b46fbbc79-4t6d8.53: 63137+ [1au] A? tracking.domain.api.int.dev-godaddy.com. (80)
21:39:35.304572 IP coredns-7b46fbbc79-4t6d8.35625 > ip-192-168-0-2.us-west-2.compute.internal.53: 14210+ [1au] A? tracking.domain.api.int.dev-godaddy.com. (80)
21:39:35.304775 IP ip-192-168-0-2.us-west-2.compute.internal.53 > coredns-7b46fbbc79-4t6d8.35625: 14210 NXDomain 0/0/1 (68)
21:39:35.304854 ARP, Request who-has ip-192-168-145-95.us-west-2.compute.internal tell coredns-7b46fbbc79-4t6d8, length 28
21:39:35.623320 ARP, Reply ip-192-168-145-95.us-west-2.compute.internal is-at 06:45:d6:fb:8d:89 (oui Unknown), length 28
21:39:35.623325 IP coredns-7b46fbbc79-4t6d8.53 > ip-192-168-145-95.us-west-2.compute.internal.54962: 63137 NXDomain 0/0/1 (68)
proxy_delay delay is documented in https://man7.org/linux/man-pages/man7/arp.7.html.e.g. Calico have disabled proxy_delay:
https://github.com/projectcalico/calico/blob/0e9c2865822eb48a7049d1933d37fc0316a9c2aa/cni-plugin/pkg/dataplane/linux/dataplane_linux.go#L394
proxy_delay on the bridge should be 0, since the default delay is not appropriate, i don't think there is a valid use case to have such delay on our bridge
proxy_delay on the bridge is 80
in an EKS-fargate pod.
Today we can specify UserName for branch vpc-branch-eni plugin, the plugin will do a look up for the user and determine the uid, then change the owner of the tap device to that uid.
But for our use case we'd like to specify UID and GID directly, can we introduce new config like UID and GID and set the ownership of tap device accordingly? The implementation is quite straightforward and I am happy to submit a PR for it.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.