Summary
Seeing a couple of leaked endpoints (possibly) when I am scaling the deployment and before the pod goes to Running state, I am scaling down the deployment.
Observed Behavior
Add Request for pod "win-webserver-75bc4c4c6f-chxlz" with container ID "9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b"
563 2020-08-06T01:34:12Z [INFO] Executing ADD with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[10.100.0.10] Domain: Search:[default.svc.cluster.local svc.cluster.local cluster.local] Options:[]}} ENIName: ENIMACAddress:02:cb:e2:d2:e5:e7 ENIIPAddress:192.168.1.161/19 VPCCIDRs:[{IP:192.168 .0.0 Mask:ffff0000}] BridgeType:L3 BridgeNetNSPath: IPAddress:192.168.28.156/19 GatewayIPAddress:192.168.0.1 InterfaceType:veth TapUserID:0 Kubernetes:{Namespace:default PodName:win-webserver-75bc4c4c6f-chxlz PodInfraContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b ServiceCIDR:10.100.0.0/16}} ContainerID:9b1eb5e026f3e03959fd1263 6265d8ba19718453d8b1642e5a014874b3e47d1b Netns:none IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=win-webserver-75bc4c4c6f-chxlz;K8S_POD_INFRA_CONTAINER_ID=9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
Created HNS Endpoint for the Pod
540 2020-08-06T01:34:11Z [INFO] Creating HNS endpoint: {"Name":"cid-9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b","VirtualNetworkName":"vpcbr02cbe2d2e5e7","Policies":[{"Type":"OutBoundNAT","ExceptionList":["192.168.0.0/16","10.100.0.0/16"]},{"Type":"ROUTE","DestinationPrefix":"10.100.0.0/16","NeedEncap":true},{"Type":"ROUTE","DestinationPre fix":"192.168.1.161/32","NeedEncap":true}],"IPAddress":"192.168.28.156","DNSSuffix":"default.svc.cluster.local,svc.cluster.local,cluster.local","DNSServerList":"10.100.0.10","PrefixLength":19}
541 2020-08-06T01:34:11Z [INFO] Received HNS endpoint response: &{Id:EFDCCA66-44F3-4F61-BC64-C8DD620DD46E Name:cid-9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b VirtualNetwork:AEC44415-A934-4370-A616-3F8E0581FB2E VirtualNetworkName:vpcbr02cbe2d2e5e7 Policies:[[123 34 69 120 99 101 112 116 105 111 110 76 105 115 116 34 58 91 34 49 57 50 46 49 54 56 46 48 46 48 47 49 54 34 44 34 49 48 46 49 48 48 46 48 46 48 47 49 54 34 93 44 34 84 121 112 101 34 58 34 79 117 116 66 111 117 110 100 78 65 84 34 125] [123 34 68 101 115 116 105 110 97 116 105 111 110 80 114 101 102 105 120 34 58 34 49 48 46 49 48 48 46 48 46 48 47 49 54 34 44 34 78 101 101 100 69 110 99 97 112 34 58 116 114 117 101 44 34 84 121 11 2 101 34 58 34 82 79 85 84 69 34 125] [123 34 68 101 115 116 105 110 97 116 105 111 110 80 114 101 102 105 120 34 58 34 49 57 50 46 49 54 56 46 49 46 49 54 49 47 51 50 34 44 34 78 101 101 100 69 110 99 97 112 34 58 116 114 117 101 44 34 84 121 112 101 34 58 34 82 79 85 84 69 34 125] [123 34 84 121 112 101 34 58 34 76 50 68 114 105 118 101 114 34 125]] MacA ddress:00-15-5D-3F-AD-0B IPAddress:192.168.28.156 DNSSuffix:default.svc.cluster.local,svc.cluster.local,cluster.local DNSServerList:10.100.0.10 GatewayAddress:192.168.0.1 EnableInternalDNS:false DisableICC:false PrefixLength:19 IsRemoteEndpoint:false Namespace:<nil>}.
542 2020-08-06T01:34:11Z [INFO] Attaching HNS endpoint EFDCCA66-44F3-4F61-BC64-C8DD620DD46E to container 9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
Received the ADD request again (twice), we are checking if the HNS Endpoint exist then not to recreate it (which is the expected behavior)
Second ADD request,
2020-08-06T01:34:12Z [INFO] Executing ADD with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[10.100.0.10] Domain: Search:[default.svc.cluster.local svc.cluster.local cluster.local] Options:[]}} ENIName: ENIMACAddress:02:cb:e2:d2:e5:e7 ENIIPAddress:192.168.1.161/19 VPCCIDRs:[{IP:192.168.0.0 Mask:ffff0000}] BridgeType:L3 BridgeNetNSPath: IPAddress:192.168.28.156/19 GatewayIPAddress:192.168.0.1 InterfaceType:veth TapUserID:0 Kubernetes:{Namespace:default PodName:win-webserver-75bc4c4c6f-chxlz PodInfraContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b ServiceCIDR:10.100.0.0/16}} ContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b Netns:none IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=win-webserver-75bc4c4c6f-chxlz;K8S_POD_INFRA_CONTAINER_ID=9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
2020-08-06T01:34:13Z [INFO] Found existing HNS endpoint cid-9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
2020-08-06T01:34:13Z [INFO] HNS endpoint cid-9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b is already attached to container ID 9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
Third ADD request,
2020-08-06T01:34:15Z [INFO] Executing ADD with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[10.100.0.10] Domain: Search:[default.svc.cluster.local svc.cluster.local cluster.local] Options:[]}} ENIName: ENIMACAddress:02:cb:e2:d2:e5:e7 ENIIPAddress:192.168.1.161/19 VPCCIDRs:[{IP:192.168.0.0 Mask:ffff0000}] BridgeType:L3 BridgeNetNSPath: IPAddress:192.168.28.156/19 GatewayIPAddress:192.168.0.1 InterfaceType:veth TapUserID:0 Kubernetes:{Namespace:default PodName:win-webserver-75bc4c4c6f-chxlz PodInfraContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b ServiceCIDR:10.100.0.0/16}} ContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b Netns:none IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=win-webserver-75bc4c4c6f-chxlz;K8S_POD_INFRA_CONTAINER_ID=9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
2020-08-06T01:34:15Z [INFO] Found existing HNS endpoint cid-9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
2020-08-06T01:34:15Z [INFO] HNS endpoint cid-9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b is already attached to container ID 9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
Now the Del Command is invoked as the pod object is deleted,
2020-08-06T01:34:15Z [INFO] Executing DEL with netconfig: &{NetConf:{CNIVersion:0.3.1 Name:vpc Type:vpc-shared-eni Capabilities:map[] IPAM:{Type:} DNS:{Nameservers:[10.100.0.10] Domain: Search:[{%namespace%}.svc.cluster.local svc.cluster.local cluster.local] Options:[]}} ENIName: ENIMACAddress:02:cb:e2:d2:e5:e7 ENIIPAddress:192.168.1.161/19 VPCCIDRs:[{IP:192.168.0.0 Mask:ffff0000}] BridgeType:L3 BridgeNetNSPath: IPAddress:<nil> GatewayIPAddress:192.168.0.1 InterfaceType:veth TapUserID:0 Kubernetes:{Namespace:default PodName:win-webserver-75bc4c4c6f-chxlz PodInfraContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b ServiceCIDR:10.100.0.0/16}} ContainerID:9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b Netns:none IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=win-webserver-75bc4c4c6f-chxlz;K8S_POD_INFRA_CONTAINER_ID=9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b.
When the plugin is trying to detach the HNS Endpoint from the container we receive an error that the container no longer exists (which is strange considering the DEL command has not succeeded yet)
2020-08-06T01:34:15Z [INFO] Detaching HNS endpoint EFDCCA66-44F3-4F61-BC64-C8DD620DD46E from container 9b1eb5e026f3e03959fd12636265d8ba19718453d8b1642e5a014874b3e47d1b netns.
2020-08-06T01:34:15Z [ERROR] Failed to delete endpoint, ignoring: A virtual machine or container with the specified identifier does not exist.
Expected Behavior
I am new to CNI Plugin side of the code, but here is my understanding
- We should receive single Add request for a given container.
- The container should not be deleted before the Del Request succeeds.
However, If the above scenario can happen ideally then we should clean up the endpoint before returning the error.
From the code I can see that we don't clean up the HNS Endpoint when we get error related to container no longer exists in the detach workflow.
// Detach the HNS endpoint from the container's network namespace.
log.Infof("Detaching HNS endpoint %s from container %s netns.", hnsEndpoint.Id, ep.ContainerID)
err = hcsshim.HotDetachEndpoint(ep.ContainerID, hnsEndpoint.Id)
if err != nil {
return err
}
Build Details
Using Build with the changes from Pull Request #46 [Which resolves a totally separate issue]
Logs