Giter Site home page Giter Site logo

amazon-vpc-cni-plugins's Introduction

Amazon VPC CNI Plugins

VPC CNI plugins for Amazon ECS and Amazon EKS.

Security disclosures

If you think you’ve found a potential security issue, please do not post it in the Issues. Instead, please follow the instructions here or email AWS security directly.

License

This library is licensed under the Apache 2.0 License.

amazon-vpc-cni-plugins's People

Contributors

aaithal avatar abhipth avatar achevuru avatar amogh09 avatar bcelenza avatar edwardxf avatar elijahpepe avatar fenxiong avatar haikuoliu avatar kiranmeduri avatar klwntsingh avatar neerajvshah avatar ofiliz avatar paulyeo21 avatar rajal-amzn avatar rawahars avatar samjkon avatar sgundapu avatar suneyz avatar vsiddharth avatar yashdalfthegray avatar yhlee-aws avatar yinyic avatar yumex93 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

amazon-vpc-cni-plugins's Issues

[vpc-shared-eni] Leaked Endpoint on DEL Command when Pod is Created and immediately deleted

Summary

Seeing a couple of leaked endpoints (possibly) when I am scaling the deployment and before the pod goes to Running state, I am scaling down the deployment.

Observed Behavior

Add Request for pod "win-webserver-75bc4c4c6f-chxlz" with container ID "9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b"

563 2020-08-06T01:34:12Z [INFO] Executing ADD with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[10.100.0.10] Domain: Search:[default.svc.cluster.local svc.cluster.local cluster.local] Options:[]}} ENIName: ENIMACAddress:02:cb:e2:d2:e5:e7 ENIIPAddress:192.168.1.161/19 VPCCIDRs:[{IP:192.168    .0.0 Mask:ffff0000}] BridgeType:L3 BridgeNetNSPath: IPAddress:192.168.28.156/19 GatewayIPAddress:192.168.0.1 InterfaceType:veth TapUserID:0 Kubernetes:{Namespace:default PodName:win-webserver-75bc4c4c6f-chxlz PodInfraContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b ServiceCIDR:10.100.0.0/16}} ContainerID:9b1eb5e026f3e03959fd1263    6265d8ba19718453d8b1642e5a014874b3e47d1b Netns:none IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=win-webserver-75bc4c4c6f-chxlz;K8S_POD_INFRA_CONTAINER_ID=9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.

Created HNS Endpoint for the Pod

540 2020-08-06T01:34:11Z [INFO] Creating HNS endpoint: {"Name":"cid-9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b","VirtualNetworkName":"vpcbr02cbe2d2e5e7","Policies":[{"Type":"OutBoundNAT","ExceptionList":["192.168.0.0/16","10.100.0.0/16"]},{"Type":"ROUTE","DestinationPrefix":"10.100.0.0/16","NeedEncap":true},{"Type":"ROUTE","DestinationPre    fix":"192.168.1.161/32","NeedEncap":true}],"IPAddress":"192.168.28.156","DNSSuffix":"default.svc.cluster.local,svc.cluster.local,cluster.local","DNSServerList":"10.100.0.10","PrefixLength":19}
541 2020-08-06T01:34:11Z [INFO] Received HNS endpoint response: &{Id:EFDCCA66-44F3-4F61-BC64-C8DD620DD46E Name:cid-9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b VirtualNetwork:AEC44415-A934-4370-A616-3F8E0581FB2E VirtualNetworkName:vpcbr02cbe2d2e5e7 Policies:[[123 34 69 120 99 101 112 116 105 111 110 76 105 115 116 34 58 91 34 49 57 50 46 49     54 56 46 48 46 48 47 49 54 34 44 34 49 48 46 49 48 48 46 48 46 48 47 49 54 34 93 44 34 84 121 112 101 34 58 34 79 117 116 66 111 117 110 100 78 65 84 34 125] [123 34 68 101 115 116 105 110 97 116 105 111 110 80 114 101 102 105 120 34 58 34 49 48 46 49 48 48 46 48 46 48 47 49 54 34 44 34 78 101 101 100 69 110 99 97 112 34 58 116 114 117 101 44 34 84 121 11    2 101 34 58 34 82 79 85 84 69 34 125] [123 34 68 101 115 116 105 110 97 116 105 111 110 80 114 101 102 105 120 34 58 34 49 57 50 46 49 54 56 46 49 46 49 54 49 47 51 50 34 44 34 78 101 101 100 69 110 99 97 112 34 58 116 114 117 101 44 34 84 121 112 101 34 58 34 82 79 85 84 69 34 125] [123 34 84 121 112 101 34 58 34 76 50 68 114 105 118 101 114 34 125]] MacA    ddress:00-15-5D-3F-AD-0B IPAddress:192.168.28.156 DNSSuffix:default.svc.cluster.local,svc.cluster.local,cluster.local DNSServerList:10.100.0.10 GatewayAddress:192.168.0.1 EnableInternalDNS:false DisableICC:false PrefixLength:19 IsRemoteEndpoint:false Namespace:<nil>}.
542 2020-08-06T01:34:11Z [INFO] Attaching HNS endpoint EFDCCA66-44F3-4F61-BC64-C8DD620DD46E to container 9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.

Received the ADD request again (twice), we are checking if the HNS Endpoint exist then not to recreate it (which is the expected behavior)

Second ADD request,

2020-08-06T01:34:12Z [INFO] Executing ADD with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[10.100.0.10] Domain: Search:[default.svc.cluster.local svc.cluster.local cluster.local] Options:[]}} ENIName: ENIMACAddress:02:cb:e2:d2:e5:e7 ENIIPAddress:192.168.1.161/19 VPCCIDRs:[{IP:192.168.0.0 Mask:ffff0000}] BridgeType:L3 BridgeNetNSPath: IPAddress:192.168.28.156/19 GatewayIPAddress:192.168.0.1 InterfaceType:veth TapUserID:0 Kubernetes:{Namespace:default PodName:win-webserver-75bc4c4c6f-chxlz PodInfraContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b ServiceCIDR:10.100.0.0/16}} ContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b Netns:none IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=win-webserver-75bc4c4c6f-chxlz;K8S_POD_INFRA_CONTAINER_ID=9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
2020-08-06T01:34:13Z [INFO] Found existing HNS endpoint cid-9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
2020-08-06T01:34:13Z [INFO] HNS endpoint cid-9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b is already attached to container ID 9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.

Third ADD request,

2020-08-06T01:34:15Z [INFO] Executing ADD with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[10.100.0.10] Domain: Search:[default.svc.cluster.local svc.cluster.local cluster.local] Options:[]}} ENIName: ENIMACAddress:02:cb:e2:d2:e5:e7 ENIIPAddress:192.168.1.161/19 VPCCIDRs:[{IP:192.168.0.0 Mask:ffff0000}] BridgeType:L3 BridgeNetNSPath: IPAddress:192.168.28.156/19 GatewayIPAddress:192.168.0.1 InterfaceType:veth TapUserID:0 Kubernetes:{Namespace:default PodName:win-webserver-75bc4c4c6f-chxlz PodInfraContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b ServiceCIDR:10.100.0.0/16}} ContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b Netns:none IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=win-webserver-75bc4c4c6f-chxlz;K8S_POD_INFRA_CONTAINER_ID=9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
2020-08-06T01:34:15Z [INFO] Found existing HNS endpoint cid-9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
2020-08-06T01:34:15Z [INFO] HNS endpoint cid-9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b is already attached to container ID 9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.

Now the Del Command is invoked as the pod object is deleted,

2020-08-06T01:34:15Z [INFO] Executing DEL with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[10.100.0.10] Domain: Search:[{%namespace%}.svc.cluster.local svc.cluster.local cluster.local] Options:[]}} ENIName: ENIMACAddress:02:cb:e2:d2:e5:e7 ENIIPAddress:192.168.1.161/19 VPCCIDRs:[{IP:192.168.0.0 Mask:ffff0000}] BridgeType:L3 BridgeNetNSPath: IPAddress:<nil> GatewayIPAddress:192.168.0.1 InterfaceType:veth TapUserID:0 Kubernetes:{Namespace:default PodName:win-webserver-75bc4c4c6f-chxlz PodInfraContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b ServiceCIDR:10.100.0.0/16}} ContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b Netns:none IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=win-webserver-75bc4c4c6f-chxlz;K8S_POD_INFRA_CONTAINER_ID=9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.

When the plugin is trying to detach the HNS Endpoint from the container we receive an error that the container no longer exists (which is strange considering the DEL command has not succeeded yet)

2020-08-06T01:34:15Z [INFO] Detaching HNS endpoint EFDCCA66-44F3-4F61-BC64-C8DD620DD46E from container 9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b netns.
2020-08-06T01:34:15Z [ERROR] Failed to delete endpoint, ignoring: A virtual machine or container with the specified identifier does not exist.

Expected Behavior

I am new to CNI Plugin side of the code, but here is my understanding

  1. We should receive single Add request for a given container.
  2. The container should not be deleted before the Del Request succeeds.

However, If the above scenario can happen ideally then we should clean up the endpoint before returning the error.

From the code I can see that we don't clean up the HNS Endpoint when we get error related to container no longer exists in the detach workflow.

// Detach the HNS endpoint from the container's network namespace.
	log.Infof("Detaching HNS endpoint %s from container %s netns.", hnsEndpoint.Id, ep.ContainerID)
	err = hcsshim.HotDetachEndpoint(ep.ContainerID, hnsEndpoint.Id)
	if err != nil {
		return err
	}

Build Details

Using Build with the changes from Pull Request #46 [Which resolves a totally separate issue]

Logs

EKS node upgrade failed | aws CNI and istio cni

Summary

EKS node upgrade failing when using istio CNI plugin,

Description

EKS is working fine when I am installing aws CNI then istio with istio CNI, both are running fine with eks and if eks autoscale itself new node up successfully,
But at later stage when i trying to upgrade nodegroup the entire upgrade progress is failing, AWS CNI pod (aws-node) is failing on new upgrade node due to which node upgrade process is failing.

Expected Behavior

Observed Behavior

Environment Details

Supporting Log Snippets

default setting of `proxy_delay` on the bridge setup by shared-vpc-eni plugin causing DNS latency

Summary

shared-vpc-eni plugin enables proxy-arp on the bridge, however, the default proxy_delay(80) is used, where linux will randomly add delays before respond to ARP request. This can cause significant latency especially in DNS requests if the dns

Description

In our environment, we configured pod networking using shared-vpc-eni plugin, and pod have a route rule that will route /19 addresses with ARP resolution through the bridge setup by shared-vpc-eni plugin. The bridge is now responding the ARP request with a random delay up to 800ms.

21:39:35.304307 IP ip-192-168-145-95.us-west-2.compute.internal.54962 > coredns-7b46fbbc79-4t6d8.53: 63137+ [1au] A? tracking.domain.api.int.dev-godaddy.com. (80)
21:39:35.304572 IP coredns-7b46fbbc79-4t6d8.35625 > ip-192-168-0-2.us-west-2.compute.internal.53: 14210+ [1au] A? tracking.domain.api.int.dev-godaddy.com. (80)
21:39:35.304775 IP ip-192-168-0-2.us-west-2.compute.internal.53 > coredns-7b46fbbc79-4t6d8.35625: 14210 NXDomain 0/0/1 (68)
21:39:35.304854 ARP, Request who-has ip-192-168-145-95.us-west-2.compute.internal tell coredns-7b46fbbc79-4t6d8, length 28
21:39:35.623320 ARP, Reply ip-192-168-145-95.us-west-2.compute.internal is-at 06:45:d6:fb:8d:89 (oui Unknown), length 28
21:39:35.623325 IP coredns-7b46fbbc79-4t6d8.53 > ip-192-168-145-95.us-west-2.compute.internal.54962: 63137 NXDomain 0/0/1 (68)

e.g. Calico have disabled proxy_delay:
https://github.com/projectcalico/calico/blob/0e9c2865822eb48a7049d1933d37fc0316a9c2aa/cni-plugin/pkg/dataplane/linux/dataplane_linux.go#L394

Expected Behavior

proxy_delay on the bridge should be 0, since the default delay is not appropriate, i don't think there is a valid use case to have such delay on our bridge

Observed Behavior

proxy_delay on the bridge is 80

Environment Details

in an EKS-fargate pod.

Supporting Log Snippets

On a host with no IPv6 capabilities the plugin crashes when blocking the IMDS endpoint

Summary

Since v1.69.0 of the ECS agent the plugin crashes, with the latest version, 1.77.0, the service can start a task, but it cannot connect to the network. As per AWS recommendations we're using awsvpc mode and blocking the IMDS endpoint, which appears to be the cause of the issue.

Description

We see an error in the logs, see below, when the plugin attempts to add the blackhole route for the IPv6 address of the IMDS, this line in the code appears to be what's failing.

image

The AMI that we use for the EC2 instance does not have IPv6 enabled for security reasons.

Expected Behavior

The plugin handles the resulting error gracefully, or provides some means of disabling the IPv6 support to avoid the error.

Observed Behavior

The plugin crashes and fails to finish configuring the network routes for the task, e.g. the default route is missing from the task route table, etc.

Route table with latest agent and ECS_AWSVPC_BLOCK_IMDS=true:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
169.254.169.254 *               255.255.255.255 UH    0      0        0 *
169.254.170.2   169.254.172.1   255.255.255.255 UGH   0      0        0 ecs-eth0
169.254.172.1   169.254.172.1   255.255.255.255 UGH   0      0        0 ecs-eth0

Route table with latest agent and ECS_AWSVPC_BLOCK_IMDS=false:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         ip-10-116-128-1 0.0.0.0         UG    0      0        0 eth0
10.116.128.128  *               255.255.255.224 U     0      0        0 eth0
169.254.170.2   169.254.172.1   255.255.255.255 UGH   0      0        0 ecs-eth0
169.254.172.1   169.254.172.1   255.255.255.255 UGH   0      0        0 ecs-eth0

Route table with v1.68.2 of the agent and ECS_AWSVPC_BLOCK_IMDS=true:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         ip-10-116-128-1 0.0.0.0         UG    0      0        0 eth0
10.116.128.160  *               255.255.255.224 U     0      0        0 eth0
169.254.169.254 *               255.255.255.255 UH    0      0        0 *
169.254.170.2   169.254.172.1   255.255.255.255 UGH   0      0        0 ecs-eth0
169.254.172.1   169.254.172.1   255.255.255.255 UGH   0      0        0 ecs-eth0

Environment Details

docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.0.0+unknown)

Server:
 Containers: 3
  Running: 3
  Paused: 0
  Stopped: 0
 Images: 3
 Server Version: 20.10.25
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1e1ea6e986c6c86565bc33d52e34b81b3e2bc71f
 runc version: f19387a6bec4944c770f7668ab51c4348d9c2f38
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.197-186.748.amzn2.x86_64
 Operating System: Amazon Linux 2
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.43GiB
 Name: ip-X-X-X-X.eu-west-2.compute.internal
 ID: U742:Y4OW:URO4:EH6C:LBN5:5HMC:HHOP:LWGQ:6RIV:FTJY:KJW3:QBBQ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Specify UID and GID for vpc-branch-eni plugin

Summary

Today we can specify UserName for branch vpc-branch-eni plugin, the plugin will do a look up for the user and determine the uid, then change the owner of the tap device to that uid.

But for our use case we'd like to specify UID and GID directly, can we introduce new config like UID and GID and set the ownership of tap device accordingly? The implementation is quite straightforward and I am happy to submit a PR for it.

Description

Expected Behavior

Observed Behavior

Environment Details

Supporting Log Snippets

CNI should support containerd on Windows

Summary

When containers are provisioned through containerd on Windows (instead of docker), the CNI plugin fails
on the ADD operation.

Description

When containers are started through containerd on Windows, the ADD operation fails due to this logic here -> https://github.com/aws/amazon-vpc-cni-plugins/blob/master/plugins/vpc-shared-eni/network/bridge_windows.go#L361

It seems like a simple fix would be to just assume that the block that has the error is invoked in the case of containerd/cri usage, in which case we can probably assume that it's only being invoked on the infra container.

Expected Behavior

The CNI should ADD successfully for containers started through containerd.

Observed Behavior

The CNI fails the ADD operation for containers started through containerd.

Environment Details

Supporting Log Snippets

2021-08-26T19:03:55Z [INFO] Executing ADD with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Names
ervers:[172.20.0.10] Domain: Search:[addon-manager-logging.svc.cluster.local svc.cluster.local cluster.local] Options:[]}} ENIName: ENIMACAddress:0a:df:16:54
:db:51 ENIIPAddress:10.17.42.208/24 VPCCIDRs:[{IP:10.17.32.0 Mask:ffffe000} {IP:10.17.64.0 Mask:ffffe000} {IP:10.17.96.0 Mask:ffffe000}] BridgeType:L3 Bridge
NetNSPath: IPAddress:10.17.42.99/24 GatewayIPAddress:10.17.42.1 InterfaceType:veth TapUserID:0 Kubernetes:{Namespace:addon-manager-logging PodName:fluent-bit
-windows-tb4ds PodInfraContainerID:20fe247f82deb6d6311c0318918999fa40d4b715b61cc97f3c264e6bb1944f79 ServiceCIDR:172.20.0.0/16}} ContainerID:20fe247f82deb6d63
11c0318918999fa40d4b715b61cc97f3c264e6bb1944f79 Netns:504314d1-16b7-47e3-8cc2-b80c1510d1b5 IfName:eth0 Args:K8S_POD_INFRA_CONTAINER_ID=20fe247f82deb6d6311c03
18918999fa40d4b715b61cc97f3c264e6bb1944f79;K8S_POD_UID=fa8bdee8-3c26-4657-8b7e-b76a1fda5601;IgnoreUnknown=1;K8S_POD_NAMESPACE=addon-manager-logging;K8S_POD_N
AME=fluent-bit-windows-tb4ds.
2021-08-26T19:03:55Z [INFO] Running on HNS version: {Major:9 Minor:5}
2021-08-26T19:03:55Z [INFO] Found existing HNS network vpcbr0adf1654db51.
2021-08-26T19:03:55Z [ERROR] Failed to parse netns 504314d1-16b7-47e3-8cc2-b80c1510d1b5 of container 20fe247f82deb6d6311c0318918999fa40d4b715b61cc97f3c264e6b
b1944f79

IP Allocation Issues

Hello. We have been seeing an issue intermittently where pods will become stuck in container creating with the following error or similar:

Warning  FailedCreatePodSandBox  56m                   kubelet, ip-10-209-103-40.us-west-2.compute.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "d20eb84d0be212c52e734d4c034f9ea29fb3a98c78d4d4cb111cf536f45298ea" network for pod "<redacted>": networkPlugin cni failed to set up pod "<redacted>" network: hnsCall failed in Win32: An address provided is invalid or reserved. (0x803b002f)

K8s version: 1.16
This pod is being scheduled on a Windows Worker Node using the amazon/Windows_Server-1909-English-Core-EKS_Optimized-1.16-2020.06.10 AMI.

We have also observed errors in the CNI logs as follows:

**2020-07-22T19:59:53Z [ERROR] Failed to parse netconfig from args: failed to parse Kubernetes args: failed to get pod settingsapi-test-connection: pods "settingsapi-test-connection" not found.
2020-07-22T19:59:53Z [ERROR] CNI command failed: failed to parse Kubernetes args: failed to get pod settingsapi-test-connection: pods "settingsapi-test-connection" not found
2020-07-22T19:59:53Z [INFO] Plugin vpc-shared-eni version  executing CNI command.
2020-07-22T19:59:53Z [ERROR] Failed to parse netconfig from args: failed to parse Kubernetes args: failed to get pod tmp-shelly3: pods "tmp-shelly3" not found.
2020-07-22T19:59:53Z [ERROR] CNI command failed: failed to parse Kubernetes args: failed to get pod tmp-shelly3: pods "tmp-shelly3" not found
2020-07-22T19:59:53Z [INFO] Plugin vpc-shared-eni version  executing CNI command.
2020-07-22T19:59:53Z [ERROR] Failed to parse netconfig from args: failed to parse Kubernetes args: failed to get pod <redacted>: pods "<redacted>" not found.
2020-07-22T19:59:53Z [ERROR] CNI command failed: failed to parse Kubernetes args: failed to get pod <redacted>: pods "<redacted>" not found
2020-07-22T19:59:58Z [INFO] Plugin vpc-shared-eni version  executing CNI command.
PS C:\ProgramData\Amazon\EKS\logs>
**

These errors appear to be referencing pods that no longer exist (and haven't for some time), are repeated constantly, and do not seem to directly correlate in start time to the incident above. The only way to stop the CNI from logging this second set errors seems to be to terminate the node, they even persist through reboot.

I'm happy to provide any more details that I can if needed.

dnsConfig are ignored for Windows pods

I'm running EKS K8S 1.30 with EC2 Windows workers nodes only, coredns runing on fargate
There is an issue with dnsConfig that are ignored for Windows pods:

    spec:
      dnsPolicy: "None"
      dnsConfig:
        nameservers:
          - 192.168.1.100
        searches:
          - custom.domain

C:\app>ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : eks-app
   Primary Dns Suffix  . . . . . . . : custom.domain
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : custom.namespace.svc.cluster.local
                                       svc.cluster.local
                                       cluster.local

Ethernet adapter vEthernet (cid-54890e7f-6a27-434b-ab4c-12ba9c74c1e6):

   Connection-specific DNS Suffix  . : custom.namespace.svc.cluster.locall
   Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Container Adapter
   Physical Address. . . . . . . . . : 00-15-5D-21-52-E4
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::e67b:4cd8:d426:be86%18(Preferred)
   IPv4 Address. . . . . . . . . . . : 10.50.130.207(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 10.50.130.1
   DNS Servers . . . . . . . . . . . : 172.20.0.10
   NetBIOS over Tcpip. . . . . . . . : Disabled
   Connection-specific DNS Suffix Search List :
                                       custom.namespace.svc.cluster.local
                                       svc.cluster.local
                                       cluster.local

Expected to happen:

Windows IP Configuration

   Host Name . . . . . . . . . . . . : eks-app
   Primary Dns Suffix  . . . . . . . : custom.domain
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : **custom.domain**

Ethernet adapter vEthernet (cid-54890e7f-6a27-434b-ab4c-12ba9c74c1e6):

   Connection-specific DNS Suffix  . : **custom.domain**
   Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Container Adapter
   Physical Address. . . . . . . . . : 00-15-5D-21-52-E4
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::e67b:4cd8:d426:be86%18(Preferred)
   IPv4 Address. . . . . . . . . . . : 10.50.130.207(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . : 10.50.130.1
   DNS Servers . . . . . . . . . . . : **192.168.1.100**
   NetBIOS over Tcpip. . . . . . . . : Disabled
   Connection-specific DNS Suffix Search List :
                                       **custom.domain**

EC2 Metadata not in sync is causing aws-node be stuck in Crash Loopback state.

IPAMD is restarting with the below messages.

{"level":"debug","ts":"2020-11-21T12:18:10.565Z","caller":"awsutils/awsutils.go:388","msg":"Update ENI eni-xxxxxxxxxx"}
{"level":"error","ts":"2020-11-21T12:18:10.773Z","caller":"aws-k8s-agent/main.go:28","msg":"Initialization failure: ipamd: can not initialize with AWS SDK interface: refreshSGIDs: unable to update the ENI's SG: InvalidNetworkInterfaceID.NotFound: The networkInterface ID 'eni-xxxxxxxxxxxx' does not exist\n\tstatus code: 400, request id: aaaaaa-bbbbbb-cccccc-ccccc-sssssss"}

From awsutils l458 We understand ModifyNetworkInterfaceAttribute action is triggering the above error message.

From the Cloud trail API calls I was able to confirm that the ENI was created and deleted by VPC cni however. They were not cleared from EC2 metadata. Since metadataMACPath function will use the below snippet to make the metadata call which will gather the eni-id We are seeing the above issue.

eniID, err = cache.ec2Metadata.GetMetadata( + eniMAC + metadataInterface)

I have confirmed by running the below curl call that the deleted ENI is still present on the metadata.

curl 169.254.169.254/2020-10-27/meta-data/network/interfaces/macs/<macaddress>/<interface-id>

Summary

IPAMD should be able to handle out-of-synced IMDS in above cases. As This will cause aws-node to be in crashloopback state and cause pods scheduling onto the node to not have any ip's causing them to be stuck in creating state.

similar issue form the past #1177

Why is this needed:
CNI should handle similar issue related to imds gracefully instead of getting the node and all pods stuck.

NetworkPolicy based on NamespaceSelector (namespace-isolation with calico) breaks windows pod communication

Summary

When I enable a policy to allow pods in one namespace to communicate with services in the same namespace windows pods can no longer communicate with any service inside that namespace. The rest of the cluster (linux nodes and pods) work as expected and can communicate.

Description

After a lot of trial and error I could pin point the issue on the fact, that the traffic from the pods on the windows node is not routed directly but NATed through the host. (I documented a lot of the process in this issue I created on the calico project projectcalico/calico#4936)
As a workaround I whitelisted the IP of the windows host in one of the namespaces which is very ugly and not really scalable.

As a start it would be nice to know how you guys intended this to function since I can't really find any useful documentation specifying what the actual expected behaviour is.

Expected Behavior

For running windows nodes in EKS, I expect kube-proxy to route the traffic directly to the VPC as is the case for the rest of the cluster running on linux nodes.

Observed Behavior

Kube-proxy on windows applies NAT to the traffic emerging from the pods.

Environment Details

EKS with k8s 1.19
Calico version 3.20
WindowsServer2019FullContainer AMI for the windows nodes
Linux nodes are managed

AppMesh iptables Proxy config?

Apologies if this isn't the correct place to ask such a question.

Given an ECS container instance with an AppMesh-enabled task (an application container and an Envoy proxy sidecar), how can I view the iptables rules created in https://github.com/aws/amazon-vpc-cni-plugins/blob/master/plugins/aws-appmesh/plugin/commands.go? I'm looking to verify these rules using the iptables command.

I don't see them on the host, or in any of the Docker network namespaces for the application or Envoy containers. Any tips would be much appreciated!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.