Repository for tools and artifacts related to the sig-windows charter in Kubernetes. Scripts to assist kubeadm and wincat and flannel will be hosted here.
License: Apache License 2.0
Repository for tools and artifacts related to the sig-windows charter in Kubernetes. Scripts to assist kubeadm and wincat and flannel will be hosted here.
Learn how to engage with the Kubernetes community on the community page.
You can reach the maintainers of this project at:
Participation in the Kubernetes community is governed by the Kubernetes Code of Conduct.
Hi guys,
This is a little difficult to explain, but it stands clear to me what the problem is. I need help to resolve the issue; i simply cannot put the pieces together in the puzzle.
OK.
I have recently, for development and testing purposes, setup a K8S setup running 1.18.3.
Flannel is 0.12. Flannel and Kube-Proxy are the modified Docker images to support Windows 1909. Docker version is 19.03.5.
I was unable to get overlay networking to work, so i switched to L2Bridge which just works.
All is actually working and I am happy camper - UNTIL i look in the logs; they are flooded with errors like these:
E0525 20:44:42.834922 9256 remote_runtime.go:495] ListContainerStats with filter &ContainerStatsFilter{Id:,PodSandboxId:,LabelSelector:map[string]string{},} from runtime service failed: rpc error: code = Unknown desc = hcsshim::OpenComputeSystem 00c1c9134ef7309ebc425c4328163bf1fd7ba5c50733e8c042e30b457c0ac99e: A virtual machine or container with the specified identifier does not exist. E0525 20:44:42.834922 9256 eviction_manager.go:255] eviction manager: failed to get summary stats: failed to list pod stats: failed to list all container stats: rpc error: code = Unknown desc = hcsshim::OpenComputeSystem 00c1c9134ef7309ebc425c4328163bf1fd7ba5c50733e8c042e30b457c0ac99e: A virtual machine or container with the specified identifier does not exist.
I figured out the reason by doing a docker ps --all. It turns out, that the referenced container id, 00c1c9134ef7309ebc425c4328163bf1fd7ba5c50733e8c042e30b457c0ac99e, has been replaced with another because of an error when the Docker was executed.
I will try to show what happened:
Container id 00c1c9134ef7309ebc425c4328163bf1fd7ba5c50733e8c042e30b457c0ac99e is a result of this:
docker inspect 00c1c9134ef7
{ "Id": "00c1c9134ef7309ebc425c4328163bf1fd7ba5c50733e8c042e30b457c0ac99e", "Created": "2020-05-24T15:23:42.4261928Z", "Path": "powershell", "Args": [ "-file", "/etc/kube-flannel-windows/run.ps1" ], "State": { "Status": "created", "Running": false, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 0, "ExitCode": 128, "Error": "hcsshim::CreateComputeSystem 00c1c9134ef7309ebc425c4328163bf1fd7ba5c50733e8c042e30b457c0ac99e: The endpoint was not found.\n(extra info: {\"SystemType\":\"Container\",\"Name\":\"00c1c9134ef7309ebc425c4328163bf1fd7ba5c50733e8c042e30b457c0ac99e\",\"Owner\":\"docker\",\"Vol umePath\":\"\\\\\\\\?\\\\Volume{88453e19-4aef-498b-9d9a-df7b3290cb1b}\",\"IgnoreFlushesDuringBoot\":true,\"LayerFolderPath\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\00c1c9134ef7309ebc425c4328163bf1fd7ba5c50733e8c042e30b457c0ac99e\",\"Layers\":[{\"ID\":\"d63dc737-8ddf-5806-903e-4bff1 b5c2c29\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\e2f6212c24460e553076add6524d7822e742aa4e747c99377f1a0847331233ae\"},{\"ID\":\"fc1a72b7-f827-5dec-aef6-3391238b7f85\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\d7f7b03b814f318086f64fbfd553262184683dbb415ee732 679180ce9589655a\"},{\"ID\":\"6ef1a610-93c0-5fd4-b7dd-b92ae69370bb\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\d319813bc01c9f0e9594e3cacd845fb2fb06fa5d4a1a3f9ed170178613eb9a8e\"},{\"ID\":\"a4aa5c9c-d609-5011-9308-8051b55b8e4d\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\window sfilter\\\\0493fb95dd006b04c5dc317770a2bd2ca9235cedca8caf53731e4d27431470fe\"},{\"ID\":\"080aac09-eb12-58f9-a0a9-a74cc24e38c5\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\43bbffe6e0d3f5732d64db164cd744248d0191f36c0aea90e050a32652809bf5\"},{\"ID\":\"a24b2d98-225b-5811-b480-815c 2ff85f59\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\67fa08a687898cdb6787055a8e0aa07998f5c3ff840feccda196bb90c3dc84d4\"},{\"ID\":\"56ac4bf8-a901-5c8f-995c-44d856523fbc\",\"Path\":\"C:\\\\ProgramData\\\\docker\\\\windowsfilter\\\\05e370e423fd314a5c0af9a11791fea4dceb61d7e04bd23 9116a057e8a093f96\"}],\"HostName\":\"ebfa0c4b7ba2\",\"MappedDirectories\":[{\"HostPath\":\"c:\\\\\",\"ContainerPath\":\"c:\\\\host\",\"ReadOnly\":false,\"BandwidthMaximum\":0,\"IOPSMaximum\":0,\"CreateInUtilityVM\":false},{\"HostPath\":\"c:\\\\var\\\\lib\\\\kubelet\\\\pods\\\\040dde66-5c63-44 53-9388-845ea450a2d3\\\\volumes\\\\kubernetes.io~configmap\\\\kube-proxy\",\"ContainerPath\":\"c:\\\\kube-proxy\",\"ReadOnly\":true,\"BandwidthMaximum\":0,\"IOPSMaximum\":0,\"CreateInUtilityVM\":false},{\"HostPath\":\"c:\\\\var\\\\lib\\\\kubelet\\\\pods\\\\040dde66-5c63-4453-9388-845ea450a2d3 \\\\volumes\\\\kubernetes.io~configmap\\\\flannel-cfg\",\"ContainerPath\":\"c:\\\\etc\\\\kube-flannel\",\"ReadOnly\":true,\"BandwidthMaximum\":0,\"IOPSMaximum\":0,\"CreateInUtilityVM\":false},{\"HostPath\":\"c:\\\\var\\\\lib\\\\kubelet\\\\pods\\\\040dde66-5c63-4453-9388-845ea450a2d3\\\\volume s\\\\kubernetes.io~configmap\\\\flannel-windows-cfg\",\"ContainerPath\":\"c:\\\\etc\\\\kube-flannel-windows\",\"ReadOnly\":true,\"BandwidthMaximum\":0,\"IOPSMaximum\":0,\"CreateInUtilityVM\":false},{\"HostPath\":\"c:\\\\var\\\\lib\\\\kubelet\\\\pods\\\\040dde66-5c63-4453-9388-845ea450a2d3\\\\ volumes\\\\kubernetes.io~configmap\\\\kubeadm-config\",\"ContainerPath\":\"c:\\\\etc\\\\kubeadm-config\",\"ReadOnly\":true,\"BandwidthMaximum\":0,\"IOPSMaximum\":0,\"CreateInUtilityVM\":false},{\"HostPath\":\"c:\\\\etc\",\"ContainerPath\":\"c:\\\\etc\\\\cni\\\\net.d\",\"ReadOnly\":false,\"Ban dwidthMaximum\":0,\"IOPSMaximum\":0,\"CreateInUtilityVM\":false},{\"HostPath\":\"c:\\\\var\\\\lib\\\\kubelet\\\\pods\\\\040dde66-5c63-4453-9388-845ea450a2d3\\\\volumes\\\\kubernetes.io~secret\\\\flannel-token-l7k68\",\"ContainerPath\":\"c:\\\\var\\\\run\\\\secrets\\\\kubernetes.io\\\\servicea ccount\",\"ReadOnly\":true,\"BandwidthMaximum\":0,\"IOPSMaximum\":0,\"CreateInUtilityVM\":false}],\"MappedPipes\":[{\"HostPath\":\"\\\\\\\\.\\\\pipe\\\\rancher_wins\",\"ContainerPipeName\":\"rancher_wins\"}],\"HvPartition\":false,\"NetworkSharedContainerName\":\"ebfa0c4b7ba22a087fd88171ea6bcf 06d765175e21aaa6fd626fd333154fc834\",\"EndpointList\":[\"B903C54E-ABE1-4C3E-8036-B12604DC7022\"]})", "StartedAt": "0001-01-01T00:00:00Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Image": "sha256:628dd779fb7f1065314b8992311c102a3f503ad70d60637c9e4e359efed5fbee", "ResolvConfPath": "", "HostnamePath": "", "HostsPath": "", "LogPath": "C:\\ProgramData\\docker\\containers\\00c1c9134ef7309ebc425c4328163bf1fd7ba5c50733e8c042e30b457c0ac99e\\00c1c9134ef7309ebc425c4328163bf1fd7ba5c50733e8c042e30b457c0ac99e-json.log", "Name": "/k8s_kube-flannel_kube-flannel-ds-windows-amd64-bp967_kube-system_040dde66-5c63-4453-9388-845ea450a2d3_9", "RestartCount": 0, "Driver": "windowsfilter", "Platform": "windows", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "", "ExecIDs": null, "HostConfig": { "Binds": [ "\\\\.\\pipe\\rancher_wins:\\\\.\\pipe\\rancher_wins", "c:/:c:/host", "c:\\var\\lib\\kubelet\\pods\\040dde66-5c63-4453-9388-845ea450a2d3\\volumes\\kubernetes.io~configmap\\kube-proxy:c:/kube-proxy:ro", "c:/etc:c:/etc/cni/net.d", "c:\\var\\lib\\kubelet\\pods\\040dde66-5c63-4453-9388-845ea450a2d3\\volumes\\kubernetes.io~configmap\\flannel-cfg:c:/etc/kube-flannel/:ro", "c:\\var\\lib\\kubelet\\pods\\040dde66-5c63-4453-9388-845ea450a2d3\\volumes\\kubernetes.io~configmap\\flannel-windows-cfg:c:/etc/kube-flannel-windows/:ro", "c:\\var\\lib\\kubelet\\pods\\040dde66-5c63-4453-9388-845ea450a2d3\\volumes\\kubernetes.io~configmap\\kubeadm-config:c:/etc/kubeadm-config/:ro", "c:\\var\\lib\\kubelet\\pods\\040dde66-5c63-4453-9388-845ea450a2d3\\volumes\\kubernetes.io~secret\\flannel-token-l7k68:c:/var/run/secrets/kubernetes.io/serviceaccount:ro" ], "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "container:ebfa0c4b7ba22a087fd88171ea6bcf06d765175e21aaa6fd626fd333154fc834", "PortBindings": null, "RestartPolicy": { "Name": "no", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "Capabilities": null, "Dns": [], "DnsOptions": [], "DnsSearch": [], "ExtraHosts": null, "GroupAdd": null, "IpcMode": "container:ebfa0c4b7ba22a087fd88171ea6bcf06d765175e21aaa6fd626fd333154fc834", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "ConsoleSize": [ 0, 0 ], "Isolation": "process", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [], "DeviceCgroupRules": null, "DeviceRequests": null, "KernelMemory": 0, "KernelMemoryTCP": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": null, "PidsLimit": null, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": null, "ReadonlyPaths": null }, "GraphDriver": { "Data": { "dir": "C:\\ProgramData\\docker\\windowsfilter\\00c1c9134ef7309ebc425c4328163bf1fd7ba5c50733e8c042e30b457c0ac99e" }, "Name": "windowsfilter" }, "Mounts": [ { "Type": "bind", "Source": "c:\\var\\lib\\kubelet\\pods\\040dde66-5c63-4453-9388-845ea450a2d3\\volumes\\kubernetes.io~configmap\\flannel-cfg", "Destination": "c:\\etc\\kube-flannel", "Mode": "", "RW": false, "Propagation": "" }, { "Type": "bind", "Source": "c:\\var\\lib\\kubelet\\pods\\040dde66-5c63-4453-9388-845ea450a2d3\\volumes\\kubernetes.io~configmap\\flannel-windows-cfg", "Destination": "c:\\etc\\kube-flannel-windows", "Mode": "", "RW": false, "Propagation": "" }, { "Type": "bind", "Source": "c:\\var\\lib\\kubelet\\pods\\040dde66-5c63-4453-9388-845ea450a2d3\\volumes\\kubernetes.io~configmap\\kubeadm-config", "Destination": "c:\\etc\\kubeadm-config", "Mode": "", "RW": false, "Propagation": "" }, { "Type": "bind", "Source": "c:\\", "Destination": "c:\\host", "Mode": "", "RW": true, "Propagation": "" }, { "Type": "bind", "Source": "c:\\var\\lib\\kubelet\\pods\\040dde66-5c63-4453-9388-845ea450a2d3\\volumes\\kubernetes.io~configmap\\kube-proxy", "Destination": "c:\\kube-proxy", "Mode": "", "RW": false, "Propagation": "" }, { "Type": "bind", "Source": "c:\\var\\lib\\kubelet\\pods\\040dde66-5c63-4453-9388-845ea450a2d3\\volumes\\kubernetes.io~secret\\flannel-token-l7k68", "Destination": "c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount", "Mode": "", "RW": false, "Propagation": "" }, { "Type": "npipe", "Source": "\\\\.\\pipe\\rancher_wins", "Destination": "\\\\.\\pipe\\rancher_wins", "Mode": "", "RW": true, "Propagation": "" }, { "Type": "bind", "Source": "c:\\etc", "Destination": "c:\\etc\\cni\\net.d", "Mode": "", "RW": true, "Propagation": "" } ], "Config": { "Hostname": "ebfa0c4b7ba2", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "POD_NAME=kube-flannel-ds-windows-amd64-bp967", "POD_NAMESPACE=kube-system", "KUBE_DNS_PORT_53_TCP_PORT=53", "KUBE_DNS_PORT_53_TCP_ADDR=10.96.0.10", "KUBERNETES_SERVICE_PORT=443", "KUBERNETES_SERVICE_PORT_HTTPS=443", "KUBERNETES_PORT=tcp://10.96.0.1:443", "KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443", "KUBE_DNS_PORT=udp://10.96.0.10:53", "KUBE_DNS_PORT_53_UDP_PORT=53", "KUBE_DNS_SERVICE_PORT_METRICS=9153", "KUBE_DNS_PORT_53_UDP=udp://10.96.0.10:53", "KUBE_DNS_PORT_53_TCP_PROTO=tcp", "KUBE_DNS_PORT_9153_TCP_PROTO=tcp", "KUBE_DNS_PORT_9153_TCP_ADDR=10.96.0.10", "KUBERNETES_PORT_443_TCP_PROTO=tcp", "KUBE_DNS_SERVICE_PORT=53", "KUBE_DNS_SERVICE_PORT_DNS_TCP=53", "KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1", "KUBE_DNS_PORT_53_UDP_PROTO=udp", "KUBE_DNS_PORT_53_UDP_ADDR=10.96.0.10", "KUBE_DNS_PORT_53_TCP=tcp://10.96.0.10:53", "KUBE_DNS_PORT_9153_TCP=tcp://10.96.0.10:9153", "KUBE_DNS_PORT_9153_TCP_PORT=9153", "KUBE_DNS_SERVICE_HOST=10.96.0.10", "KUBE_DNS_SERVICE_PORT_DNS=53", "KUBERNETES_SERVICE_HOST=10.96.0.1", "KUBERNETES_PORT_443_TCP_PORT=443" ], "Cmd": [ "-file", "/etc/kube-flannel-windows/run.ps1" ], "Test": [ "NONE" ] }, "Image": "sha256:628dd779fb7f1065314b8992311c102a3f503ad70d60637c9e4e359efed5fbee", "Volumes": null, "WorkingDir": "", "Entrypoint": [ "powershell" ], "OnBuild": null, "Labels": { "annotation.io.kubernetes.container.hash": "2771711d", "annotation.io.kubernetes.container.restartCount": "9", "annotation.io.kubernetes.container.terminationMessagePath": "/dev/termination-log", "annotation.io.kubernetes.container.terminationMessagePolicy": "File", "annotation.io.kubernetes.pod.terminationGracePeriod": "30", "io.kubernetes.container.logpath": "\\var\\log\\pods\\kube-system_kube-flannel-ds-windows-amd64-bp967_040dde66-5c63-4453-9388-845ea450a2d3\\kube-flannel\\9.log", "io.kubernetes.container.name": "kube-flannel", "io.kubernetes.docker.type": "container", "io.kubernetes.pod.name": "kube-flannel-ds-windows-amd64-bp967", "io.kubernetes.pod.namespace": "kube-system", "io.kubernetes.pod.uid": "040dde66-5c63-4453-9388-845ea450a2d3", "io.kubernetes.sandbox.id": "ebfa0c4b7ba22a087fd88171ea6bcf06d765175e21aaa6fd626fd333154fc834" } }, "NetworkSettings": { "Bridge": "", "SandboxID": "", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": {}, "SandboxKey": "", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": {} } }
However, this no longer exists, as the system spun up a new flannel. This is described here:
docker inspect 930fc600052e
{ "Id": "930fc600052e53c62dd62f27d7a1f5e31cd9737fd5ded949f6511a2d47b4e37d", "Created": "2020-05-24T15:24:25.8159091Z", "Path": "powershell", "Args": [ "-file", "/etc/kube-flannel-windows/run.ps1" ], "State": { "Status": "running", "Running": true, "Paused": false, "Restarting": false, "OOMKilled": false, "Dead": false, "Pid": 7748, "ExitCode": 0, "Error": "", "StartedAt": "2020-05-24T15:24:27.9507771Z", "FinishedAt": "0001-01-01T00:00:00Z" }, "Image": "sha256:628dd779fb7f1065314b8992311c102a3f503ad70d60637c9e4e359efed5fbee", "ResolvConfPath": "", "HostnamePath": "", "HostsPath": "", "LogPath": "C:\\ProgramData\\docker\\containers\\930fc600052e53c62dd62f27d7a1f5e31cd9737fd5ded949f6511a2d47b4e37d\\930fc600052e53c62dd62f27d7a1f5e31cd9737fd5ded949f6511a2d47b4e37d-json.log", "Name": "/k8s_kube-flannel_kube-flannel-ds-windows-amd64-bp967_kube-system_040dde66-5c63-4453-9388-845ea450a2d3_10", "RestartCount": 0, "Driver": "windowsfilter", "Platform": "windows", "MountLabel": "", "ProcessLabel": "", "AppArmorProfile": "", "ExecIDs": null, "HostConfig": { "Binds": [ "\\\\.\\pipe\\rancher_wins:\\\\.\\pipe\\rancher_wins", "c:/:c:/host", "c:\\var\\lib\\kubelet\\pods\\040dde66-5c63-4453-9388-845ea450a2d3\\volumes\\kubernetes.io~configmap\\kube-proxy:c:/kube-proxy:ro", "c:/etc:c:/etc/cni/net.d", "c:\\var\\lib\\kubelet\\pods\\040dde66-5c63-4453-9388-845ea450a2d3\\volumes\\kubernetes.io~configmap\\flannel-cfg:c:/etc/kube-flannel/:ro", "c:\\var\\lib\\kubelet\\pods\\040dde66-5c63-4453-9388-845ea450a2d3\\volumes\\kubernetes.io~configmap\\flannel-windows-cfg:c:/etc/kube-flannel-windows/:ro", "c:\\var\\lib\\kubelet\\pods\\040dde66-5c63-4453-9388-845ea450a2d3\\volumes\\kubernetes.io~configmap\\kubeadm-config:c:/etc/kubeadm-config/:ro", "c:\\var\\lib\\kubelet\\pods\\040dde66-5c63-4453-9388-845ea450a2d3\\volumes\\kubernetes.io~secret\\flannel-token-l7k68:c:/var/run/secrets/kubernetes.io/serviceaccount:ro" ], "ContainerIDFile": "", "LogConfig": { "Type": "json-file", "Config": {} }, "NetworkMode": "container:d38522f012226c15fc27e0fd68a9ab4502c4d10feab1db72b098293663d56032", "PortBindings": null, "RestartPolicy": { "Name": "no", "MaximumRetryCount": 0 }, "AutoRemove": false, "VolumeDriver": "", "VolumesFrom": null, "CapAdd": null, "CapDrop": null, "Capabilities": null, "Dns": null, "DnsOptions": null, "DnsSearch": null, "ExtraHosts": null, "GroupAdd": null, "IpcMode": "container:d38522f012226c15fc27e0fd68a9ab4502c4d10feab1db72b098293663d56032", "Cgroup": "", "Links": null, "OomScoreAdj": 0, "PidMode": "", "Privileged": false, "PublishAllPorts": false, "ReadonlyRootfs": false, "SecurityOpt": null, "UTSMode": "", "UsernsMode": "", "ShmSize": 67108864, "ConsoleSize": [ 0, 0 ], "Isolation": "process", "CpuShares": 0, "Memory": 0, "NanoCpus": 0, "CgroupParent": "", "BlkioWeight": 0, "BlkioWeightDevice": null, "BlkioDeviceReadBps": null, "BlkioDeviceWriteBps": null, "BlkioDeviceReadIOps": null, "BlkioDeviceWriteIOps": null, "CpuPeriod": 0, "CpuQuota": 0, "CpuRealtimePeriod": 0, "CpuRealtimeRuntime": 0, "CpusetCpus": "", "CpusetMems": "", "Devices": [], "DeviceCgroupRules": null, "DeviceRequests": null, "KernelMemory": 0, "KernelMemoryTCP": 0, "MemoryReservation": 0, "MemorySwap": 0, "MemorySwappiness": null, "OomKillDisable": null, "PidsLimit": null, "Ulimits": null, "CpuCount": 0, "CpuPercent": 0, "IOMaximumIOps": 0, "IOMaximumBandwidth": 0, "MaskedPaths": null, "ReadonlyPaths": null }, "GraphDriver": { "Data": { "dir": "C:\\ProgramData\\docker\\windowsfilter\\930fc600052e53c62dd62f27d7a1f5e31cd9737fd5ded949f6511a2d47b4e37d" }, "Name": "windowsfilter" }, "Mounts": [ { "Type": "bind", "Source": "c:\\var\\lib\\kubelet\\pods\\040dde66-5c63-4453-9388-845ea450a2d3\\volumes\\kubernetes.io~configmap\\flannel-cfg", "Destination": "c:\\etc\\kube-flannel", "Mode": "", "RW": false, "Propagation": "" }, { "Type": "bind", "Source": "c:\\var\\lib\\kubelet\\pods\\040dde66-5c63-4453-9388-845ea450a2d3\\volumes\\kubernetes.io~configmap\\flannel-windows-cfg", "Destination": "c:\\etc\\kube-flannel-windows", "Mode": "", "RW": false, "Propagation": "" }, { "Type": "bind", "Source": "c:\\var\\lib\\kubelet\\pods\\040dde66-5c63-4453-9388-845ea450a2d3\\volumes\\kubernetes.io~configmap\\kubeadm-config", "Destination": "c:\\etc\\kubeadm-config", "Mode": "", "RW": false, "Propagation": "" }, { "Type": "bind", "Source": "c:\\var\\lib\\kubelet\\pods\\040dde66-5c63-4453-9388-845ea450a2d3\\volumes\\kubernetes.io~secret\\flannel-token-l7k68", "Destination": "c:\\var\\run\\secrets\\kubernetes.io\\serviceaccount", "Mode": "", "RW": false, "Propagation": "" }, { "Type": "npipe", "Source": "\\\\.\\pipe\\rancher_wins", "Destination": "\\\\.\\pipe\\rancher_wins", "Mode": "", "RW": true, "Propagation": "" }, { "Type": "bind", "Source": "c:\\", "Destination": "c:\\host", "Mode": "", "RW": true, "Propagation": "" }, { "Type": "bind", "Source": "c:\\var\\lib\\kubelet\\pods\\040dde66-5c63-4453-9388-845ea450a2d3\\volumes\\kubernetes.io~configmap\\kube-proxy", "Destination": "c:\\kube-proxy", "Mode": "", "RW": false, "Propagation": "" }, { "Type": "bind", "Source": "c:\\etc", "Destination": "c:\\etc\\cni\\net.d", "Mode": "", "RW": true, "Propagation": "" } ], "Config": { "Hostname": "d38522f01222", "Domainname": "", "User": "", "AttachStdin": false, "AttachStdout": false, "AttachStderr": false, "Tty": false, "OpenStdin": false, "StdinOnce": false, "Env": [ "POD_NAME=kube-flannel-ds-windows-amd64-bp967", "POD_NAMESPACE=kube-system", "KUBE_DNS_SERVICE_PORT_DNS_TCP=53", "KUBE_DNS_SERVICE_PORT_METRICS=9153", "KUBE_DNS_PORT_53_UDP_PROTO=udp", "KUBERNETES_SERVICE_HOST=10.96.0.1", "KUBERNETES_PORT_443_TCP=tcp://10.96.0.1:443", "KUBERNETES_PORT_443_TCP_ADDR=10.96.0.1", "KUBE_DNS_SERVICE_PORT=53", "KUBE_DNS_SERVICE_PORT_DNS=53", "KUBE_DNS_PORT_53_TCP_PROTO=tcp", "KUBE_DNS_PORT_53_TCP_PORT=53", "KUBE_DNS_PORT_9153_TCP_PROTO=tcp", "KUBE_DNS_PORT_9153_TCP_ADDR=10.96.0.10", "KUBERNETES_PORT=tcp://10.96.0.1:443", "KUBERNETES_PORT_443_TCP_PROTO=tcp", "KUBE_DNS_PORT=udp://10.96.0.10:53", "KUBE_DNS_PORT_9153_TCP_PORT=9153", "KUBE_DNS_PORT_9153_TCP=tcp://10.96.0.10:9153", "KUBERNETES_SERVICE_PORT_HTTPS=443", "KUBERNETES_PORT_443_TCP_PORT=443", "KUBE_DNS_PORT_53_UDP=udp://10.96.0.10:53", "KUBE_DNS_PORT_53_UDP_ADDR=10.96.0.10", "KUBE_DNS_PORT_53_TCP=tcp://10.96.0.10:53", "KUBERNETES_SERVICE_PORT=443", "KUBE_DNS_SERVICE_HOST=10.96.0.10", "KUBE_DNS_PORT_53_UDP_PORT=53", "KUBE_DNS_PORT_53_TCP_ADDR=10.96.0.10" ], "Cmd": [ "-file", "/etc/kube-flannel-windows/run.ps1" ], "Healthcheck": { "Test": [ "NONE" ] }, "Image": "sha256:628dd779fb7f1065314b8992311c102a3f503ad70d60637c9e4e359efed5fbee", "Volumes": null, "WorkingDir": "", "Entrypoint": [ "powershell" ], "OnBuild": null, "Labels": { "annotation.io.kubernetes.container.hash": "2771711d", "annotation.io.kubernetes.container.restartCount": "10", "annotation.io.kubernetes.container.terminationMessagePath": "/dev/termination-log", "annotation.io.kubernetes.container.terminationMessagePolicy": "File", "annotation.io.kubernetes.pod.terminationGracePeriod": "30", "io.kubernetes.container.logpath": "\\var\\log\\pods\\kube-system_kube-flannel-ds-windows-amd64-bp967_040dde66-5c63-4453-9388-845ea450a2d3\\kube-flannel\\10.log", "io.kubernetes.container.name": "kube-flannel", "io.kubernetes.docker.type": "container", "io.kubernetes.pod.name": "kube-flannel-ds-windows-amd64-bp967", "io.kubernetes.pod.namespace": "kube-system", "io.kubernetes.pod.uid": "040dde66-5c63-4453-9388-845ea450a2d3", "io.kubernetes.sandbox.id": "d38522f012226c15fc27e0fd68a9ab4502c4d10feab1db72b098293663d56032" } }, "NetworkSettings": { "Bridge": "", "SandboxID": "", "HairpinMode": false, "LinkLocalIPv6Address": "", "LinkLocalIPv6PrefixLen": 0, "Ports": {}, "SandboxKey": "", "SecondaryIPAddresses": null, "SecondaryIPv6Addresses": null, "EndpointID": "", "Gateway": "", "GlobalIPv6Address": "", "GlobalIPv6PrefixLen": 0, "IPAddress": "", "IPPrefixLen": 0, "IPv6Gateway": "", "MacAddress": "", "Networks": {} } }
I would expect k8s for Windows to keep track of the respectful container ids - or maybe a safer method, would be to assign a name and the refer to this?
I can see that this container is called by the flannel pod - i just can't see where i can make a change that suggest my above proposal.
Restarting kubelet makes no difference; it is still "tied" to the container that no longer exists.
Let me know if you need anymore information - thanks.
Hi,
I have a mixed cluster. The pods on linux machines are working properly but the pods scheduled on windows machine cannot reach outside world. They only can ping themselves and master node IP. They cant reach linux nodes or for example a 8.8.8.8. Dns names are resolved though.
In the flannel 0.12 there is constant error log:
Failed to list *v1.Node: Get https://xxxx:6443/api/v1/nodes?resourceVersion=0: http2: no cached connection was available\n","stream":"stderr"
Any solution to this problem?
So, after a whole weekend of trial and error with the "official" guide from Microsoft (https://docs.microsoft.com/en-us/virtualization/windowscontainers/kubernetes/network-topologies) I decided (more or less by coincidence) to give this a try (refereed from https://kubernetes.io/docs/setup/production-environment/windows/user-guide-windows-nodes/).
After my third re-install (which should have been the charm) i keep being blocked by a network issue.
It is running on a hypervisor having the en_windows_server_version_1909_updated_feb_2020_x64_dvd_8c906407.iso (Windows Server Version 1909) and an Intel NUC I7.
So, after the install, which followed this modified script:
{
"Cri" : {
"Name" : "dockerd",
"Images" : {
"Pause" : "mcr.microsoft.com/oss/kubernetes/pause:1.3.0",
"Nanoserver" : "mcr.microsoft.com/windows/nanoserver:1909",
"ServerCore" : "mcr.microsoft.com/windows/servercore:1909"
}
},
"Cni" : {
"Name" : "flannel",
"Source" : [{
"Name" : "flanneld",
"Url" : "https://github.com/coreos/flannel/releases/download/v0.11.0/flanneld.exe"
}
],
"Plugin" : {
"Name": "vxlan"
},
"InterfaceName" : "Ethernet"
},
"Kubernetes" : {
"Source" : {
"Release" : "1.17.3",
"Url" : "https://dl.k8s.io/v1.17.3/kubernetes-node-windows-amd64.tar.gz"
},
"ControlPlane" : {
"IpAddress" : "1.2.3.4",
"Username" : "a-username",
"KubeadmToken" : "a-token",
"KubeadmCAHash" : "sha256:a-256-hash"
},
"KubeProxy" : {
"Gates" : "WinOverlay=true"
},
"Network" : {
"ServiceCidr" : "10.96.0.0/12",
"ClusterCidr" : "10.244.0.0/16"
}
},
"Install" : {
"Destination" : "C:\\ProgramData\\Kubernetes"
}
}
Everything supposedly goes fine until the flanneld is firing up:
C:\ProgramData\Kubernetes\flanneld.exe FlannelD()
OnStart - Successfully started the service
I0309 23:48:55.743871 3408 main.go:450] Searching for interface using 1.2.3.46
I0309 23:48:55.898487 3408 main.go:210] Could not find valid interface matching 1.2.3.46: error looking up interface 1.2.3.46: Interface not found: 1.2.3.46
E0309 23:48:55.898487 3408 main.go:234] Failed to find interface to use that matches the interfaces and/or regexes provided
C:\ProgramData\Kubernetes\flanneld.exe exited unexpectedly 1
And the kubelet gives these errors:
Running on machine: k8s-n-w2-win
Binary: Built with gc go1.13.6 for windows/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0309 23:48:52.905969 700 server.go:749] kubelet needs to run as uid `0`. It is being run as -1
E0309 23:48:53.012433 700 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
E0309 23:48:53.257241 700 processstarttime.go:48] Could not get process start time, could not read /proc: CreateFile /proc: The system cannot find the file specified.
E0309 23:48:53.257241 700 processstarttime.go:48] Could not get process start time, could not read /proc: CreateFile /proc: The system cannot find the file specified.
E0309 23:48:53.258240 700 kubelet.go:1844] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
E0309 23:48:53.261240 700 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
E0309 23:48:53.269244 700 eviction_manager.go:246] eviction manager: failed to get summary stats: failed to get node info: node "k8s-n-w2-win" not found
E0309 23:48:53.288039 700 controller.go:227] failed to get node "k8s-n-w2-win" when trying to set owner ref to the node lease: nodes "k8s-n-w2-win" not found
E0309 23:48:53.359737 700 kubelet.go:2263] node "k8s-n-w2-win" not found
E0309 23:48:53.461269 700 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
E0309 23:48:55.908379 700 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.RuntimeClass: Get https://k8s:6443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:55.908379 700 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://k8s:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dk8s-n-w2-win&limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:55.908379 700 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.CSIDriver: Get https://k8s:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:55.908379 700 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:449: Failed to list *v1.Service: Get https://k8s:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:55.909379 700 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:458: Failed to list *v1.Node: Get https://k8s:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-n-w2-win&limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:56.908606 700 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.RuntimeClass: Get https://k8s:6443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:56.910580 700 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://k8s:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dk8s-n-w2-win&limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:56.911582 700 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:458: Failed to list *v1.Node: Get https://k8s:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-n-w2-win&limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:56.912583 700 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.CSIDriver: Get https://k8s:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:56.913588 700 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:449: Failed to list *v1.Service: Get https://k8s:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:57.916022 700 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:449: Failed to list *v1.Service: Get https://k8s:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:57.916022 700 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.RuntimeClass: Get https://k8s:6443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:57.917016 700 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: Get https://k8s:6443/api/v1/pods?fieldSelector=spec.nodeName%3Dk8s-n-w2-win&limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:57.917016 700 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:458: Failed to list *v1.Node: Get https://k8s:6443/api/v1/nodes?fieldSelector=metadata.name%3Dk8s-n-w2-win&limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:57.917016 700 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.CSIDriver: Get https://k8s:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: dial tcp: lookup k8s: getaddrinfow: The requested name is valid, but no data of the requested type was found.
E0309 23:48:58.268916 700 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
I have disabled the firewall completely.
Linux is working perfectly.
Here is a kubectl get pods -n=kube-system
coredns-6955765f44-ndcnk 1/1 Running 0 3h56m
coredns-6955765f44-zt8qj 1/1 Running 0 3h56m
etcd-k8s-m-m 1/1 Running 0 3h53m
etcd-k8s-n-m 1/1 Running 0 3h52m
etcd-k8s-t-m 1/1 Running 0 3h56m
kube-apiserver-k8s-m-m 1/1 Running 0 3h53m
kube-apiserver-k8s-n-m 1/1 Running 0 3h52m
kube-apiserver-k8s-t-m 1/1 Running 0 3h56m
kube-controller-manager-k8s-m-m 1/1 Running 1 3h53m
kube-controller-manager-k8s-n-m 1/1 Running 0 3h52m
kube-controller-manager-k8s-t-m 1/1 Running 1 3h56m
kube-flannel-ds-amd64-2ptrt 1/1 Running 0 152m
kube-flannel-ds-amd64-pb7dl 1/1 Running 1 118m
kube-flannel-ds-amd64-rmfp2 1/1 Running 0 152m
kube-flannel-ds-amd64-szpbk 1/1 Running 0 153m
kube-flannel-ds-amd64-vsdc9 1/1 Running 2 118m
kube-flannel-ds-amd64-zfnd5 1/1 Running 1 118m
kube-proxy-78w2f 1/1 Running 1 3h53m
kube-proxy-7bfzv 1/1 Running 1 118m
kube-proxy-cfl6z 1/1 Running 0 3h56m
kube-proxy-jw8kw 1/1 Running 0 3h52m
kube-proxy-m8flw 1/1 Running 1 118m
kube-proxy-qvb52 1/1 Running 1 118m
kube-scheduler-k8s-m-m 1/1 Running 0 3h53m
kube-scheduler-k8s-n-m 1/1 Running 0 3h52m
kube-scheduler-k8s-t-m 1/1 Running 1 3h56m
Here is the kube-flannel.yaml
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: psp.flannel.unprivileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
privileged: false
volumes:
- configMap
- secret
- emptyDir
- hostPath
allowedHostPaths:
- pathPrefix: "/etc/cni/net.d"
- pathPrefix: "/etc/kube-flannel"
- pathPrefix: "/run/flannel"
readOnlyRootFilesystem: false
# Users and groups
runAsUser:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
fsGroup:
rule: RunAsAny
# Privilege Escalation
allowPrivilegeEscalation: false
defaultAllowPrivilegeEscalation: false
# Capabilities
allowedCapabilities: ['NET_ADMIN']
defaultAddCapabilities: []
requiredDropCapabilities: []
# Host namespaces
hostPID: false
hostIPC: false
hostNetwork: true
hostPorts:
- min: 0
max: 65535
# SELinux
seLinux:
# SELinux is unused in CaaSP
rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
rules:
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "vxlan0",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan",
"VNI" : 4096,
"Port": 4789
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds-amd64
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: beta.kubernetes.io/os
operator: In
values:
- linux
- key: beta.kubernetes.io/arch
operator: In
values:
- amd64
hostNetwork: true
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.11.0-amd64
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.11.0-amd64
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
Any help is greatly appreciated.
When I do a kubectl get nodes, it is listing the win worker .. but given the errors listed above, i have no hope that it is working.
NAME STATUS ROLES AGE VERSION
k8s-m-m Ready master 3h31m v1.17.3
k8s-m-w1 Ready <none> 96m v1.17.3
k8s-n-m Ready master 3h30m v1.17.3
k8s-n-w1 Ready <none> 96m v1.17.3
k8s-n-w2-win Ready <none> 6m43s v1.17.3
k8s-t-m Ready master 3h34m v1.17.3
k8s-t-w1 Ready <none> 96m v1.17.3
So until i hear from a smart person, I have applied the -reset to the scripts.
Thanks.
I'm running the command: .\kubecluster.ps1 -join -configFile .\kubeclustervxlan.json
Error:
Waiting for service [FlannelD] to be running
Waiting for the Network (vxlan0) to be created by flanneld
You cannot call a method on a null-valued expression.
At C:\Users\Administrator\scripts\KubeClusterHelper.psm1:401 char:12
+ return $sourceVipJSONData.ip4.ip.Split("/")[0]
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidOperation: (:) [], RuntimeException
+ FullyQualifiedErrorId : InvokeMethodOnNull
This produces occasionally, sometimes running reset and then attempting to join again works so the issue is not very reproducible. It causes kube-proxy to not start.
SourceVip.json:
{
"code": 100,
"msg": "failed to allocate for range 0: 10.244.6.3 has been allocated to dummy, duplicate allocation is not allowed"
}
I think ever since #16 was merged and changed around the directory structure the docs at https://kubernetes.io/docs/setup/production-environment/windows/user-guide-windows-nodes/#preparing-a-windows-node are now no longer correct.
After PR #12 the script fails to join a cluster with a single master.
This is because in GetAPIServerEndpoint, $i.subsets.addresses.ip is not an array if there is one master. Because of this, the API server endpoint is returned as 1:6443 where 1 represents the first letter of the IP address. So we get the first character of the IP instead of the first IP.
Hello,
Do you provide support for sigwindowstools/kube-proxy:v1.16.11? I'm running Windows 1809 and K8s version 1.16.11. The flannel pods are running fine and my Windows node is in a Ready state. However, the pod responsible for the above image is stuck in an ErrImagePull state.
Attempting to pull the image on the host leads to this error:
Error response from daemon: manifest for sigwindowstools/kube-proxy:v1.16.11 not found: manifest unknown: manifest unknown
I've looked at Docker Hub and unfortunately I don't see a version for 1.16.11. Is that something easy to get going or are there some technical limitations? If it is possible, I may be able to contribute.
Thanks!
C:\Users\HBB>ver
Microsoft Windows [版本 10.0.17763.1158]
PS C:\Users\HBB> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
sigwindowstools/kube-proxy v1.18.1 1df35c13f175 3 weeks ago 5.04GB
192.168.8.126:5000/mcr.microsoft.com/windows/servercore ltsc2019 fdf6432edbdc 4 weeks ago 4.94GB
mcr.microsoft.com/windows/servercore ltsc2019 fdf6432edbdc 4 weeks ago 4.94GB
microsoft/nanoserver latest 716bb79b7dcd 4 weeks ago 251MB
mcr.microsoft.com/windows/nanoserver 1809 716bb79b7dcd 4 weeks ago 251MB
mcr.microsoft.com/windows/nanoserver latest 716bb79b7dcd 4 weeks ago 251MB
sigwindowstools/flannel 0.12.0 9499a92cb176 6 weeks ago 5.06GB
mcr.microsoft.com/k8s/core/pause 1.2.0 a74290a8271a 10 months ago 253MB
stefanscherer/registry-windows latest bc3ca37c7b77 15 months ago 1.19GB
<none> <none> 4eec02d003a4 15 months ago 364MB
[root@m1 k8s]# kubectl get pod -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-66bff467f8-vrpml 1/1 Running 2 4d14h
kube-system coredns-66bff467f8-ww22p 1/1 Running 2 4d14h
kube-system etcd-m1 1/1 Running 3 4d14h
kube-system kube-apiserver-m1 1/1 Running 2 4d14h
kube-system kube-controller-manager-m1 1/1 Running 3 4d14h
kube-system kube-flannel-ds-amd64-f8ph2 1/1 Running 2 4d13h
kube-system kube-flannel-ds-windows-amd64-9w5ls 1/1 CrashLoopBackOff 5 6m20s
kube-system kube-proxy-mhgsg 1/1 Running 2 4d14h
kube-system kube-proxy-windows-2lkdp 1/1 Running 2 13m
kube-system kube-scheduler-m1 1/1 Running 3 4d14h
Log file created at: 2020/05/12 12:10:28
Running on machine: w1
Binary: Built with gc go1.13.9 for windows/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0512 12:10:28.174731 6984 server.go:754] kubelet needs to run as uid `0`. It is being run as -1
E0512 12:10:28.644837 6984 file_unsupported.go:28] Watching source file is unsupported in this build
E0512 12:10:35.532536 6984 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
E0512 12:10:36.130945 6984 processstarttime.go:43] Could not get process start time, could not read /proc: CreateFile /proc: The system cannot find the file specified.
E0512 12:10:36.131945 6984 kubelet.go:1845] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
E0512 12:10:36.267935 6984 remote_runtime.go:495] ListContainerStats with filter &ContainerStatsFilter{Id:,PodSandboxId:,LabelSelector:map[string]string{},} from runtime service failed: rpc error: code = Unknown desc = hcsshim::OpenComputeSystem dc03e5b036401f652a100f923f849db5ed75d16ffac3606f824286b5530fbb84: A virtual machine or container with the specified identifier does not exist.
E0512 12:10:36.269935 6984 eviction_manager.go:255] eviction manager: failed to get summary stats: failed to list pod stats: failed to list all container stats: rpc error: code = Unknown desc = hcsshim::OpenComputeSystem dc03e5b036401f652a100f923f849db5ed75d16ffac3606f824286b5530fbb84: A virtual machine or container with the specified identifier does not exist.
E0512 12:10:36.715904 6984 cni.go:364] Error adding kube-system_kube-flannel-ds-windows-amd64-f9nxt/6c3e6c7d4cd542714d79de8eb21fe2deaa0575cebb044e0492c61df7dd3ece3e to network flannel/vxlan0: error while ProvisionEndpoint(c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0,B25F6D1F-4CFC-4E8C-9AC4-31C8B91856BA,6c3e6c7d4cd542714d79de8eb21fe2deaa0575cebb044e0492c61df7dd3ece3e): failed to find HNSEndpoint c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0: Endpoint c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0 not found
E0512 12:10:36.716905 6984 cni_windows.go:59] error while adding to cni network: error while ProvisionEndpoint(c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0,B25F6D1F-4CFC-4E8C-9AC4-31C8B91856BA,6c3e6c7d4cd542714d79de8eb21fe2deaa0575cebb044e0492c61df7dd3ece3e): failed to find HNSEndpoint c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0: Endpoint c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0 not found
E0512 12:10:37.211469 6984 cni.go:364] Error adding kube-system_kube-flannel-ds-windows-amd64-f9nxt/dc03e5b036401f652a100f923f849db5ed75d16ffac3606f824286b5530fbb84 to network flannel/vxlan0: error while ProvisionEndpoint(c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0,B25F6D1F-4CFC-4E8C-9AC4-31C8B91856BA,dc03e5b036401f652a100f923f849db5ed75d16ffac3606f824286b5530fbb84): failed to find HNSEndpoint c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0: Endpoint c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0 not found
E0512 12:10:37.212468 6984 cni_windows.go:59] error while adding to cni network: error while ProvisionEndpoint(c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0,B25F6D1F-4CFC-4E8C-9AC4-31C8B91856BA,dc03e5b036401f652a100f923f849db5ed75d16ffac3606f824286b5530fbb84): failed to find HNSEndpoint c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0: Endpoint c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0 not found
E0512 12:10:43.182116 6984 cni.go:364] Error adding kube-system_kube-flannel-ds-windows-amd64-f9nxt/6c3e6c7d4cd542714d79de8eb21fe2deaa0575cebb044e0492c61df7dd3ece3e to network flannel/vxlan0: error while ProvisionEndpoint(c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0,B25F6D1F-4CFC-4E8C-9AC4-31C8B91856BA,6c3e6c7d4cd542714d79de8eb21fe2deaa0575cebb044e0492c61df7dd3ece3e): failed to find HNSEndpoint c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0: Endpoint c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0 not found
E0512 12:10:43.183117 6984 cni_windows.go:59] error while adding to cni network: error while ProvisionEndpoint(c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0,B25F6D1F-4CFC-4E8C-9AC4-31C8B91856BA,6c3e6c7d4cd542714d79de8eb21fe2deaa0575cebb044e0492c61df7dd3ece3e): failed to find HNSEndpoint c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0: Endpoint c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0 not found
E0512 12:10:43.652266 6984 cni.go:364] Error adding kube-system_kube-flannel-ds-windows-amd64-f9nxt/dc03e5b036401f652a100f923f849db5ed75d16ffac3606f824286b5530fbb84 to network flannel/vxlan0: error while ProvisionEndpoint(c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0,B25F6D1F-4CFC-4E8C-9AC4-31C8B91856BA,dc03e5b036401f652a100f923f849db5ed75d16ffac3606f824286b5530fbb84): failed to find HNSEndpoint c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0: Endpoint c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0 not found
E0512 12:10:43.653268 6984 cni_windows.go:59] error while adding to cni network: error while ProvisionEndpoint(c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0,B25F6D1F-4CFC-4E8C-9AC4-31C8B91856BA,dc03e5b036401f652a100f923f849db5ed75d16ffac3606f824286b5530fbb84): failed to find HNSEndpoint c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0: Endpoint c106f269fa88fa7513d8a27b5e60f6868653a79c59ac0300804648c75185fdd1_vxlan0 not found
E0512 12:10:46.864372 6984 remote_runtime.go:495] ListContainerStats with filter &ContainerStatsFilter{Id:,PodSandboxId:,LabelSelector:map[string]string{},} from runtime service failed: rpc error: code = Unknown desc = hcsshim::OpenComputeSystem 6c3e6c7d4cd542714d79de8eb21fe2deaa0575cebb044e0492c61df7dd3ece3e: A virtual machine or container with the specified identifier does not exist.
.
.
.
PS C:\Users\HBB> Get-HnsNetwork
ActivityId : FA7B5842-9A55-49F9-A483-1678F853A1B0
AdditionalParams :
CurrentEndpointCount : 1
Extensions : {@{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A; IsEnabled=False; Name=Microsoft Windows 筛选平台}, @{Id=E9B59CFA-2BE1-4B21-82
8F-B6FBDBDDC017; IsEnabled=False; Name=Microsoft Azure VFP Switch Extension}, @{Id=EA24CD6C-D17A-4348-9190-09F0D5BE83DD;
IsEnabled=True; Name=Microsoft NDIS 捕获}}
Flags : 0
Health : @{AddressNotificationMissedCount=0; AddressNotificationSequenceNumber=0; InterfaceNotificationMissedCount=0; InterfaceNot
ificationSequenceNumber=0; LastErrorCode=0; LastUpdateTime=132337283826313062; RouteNotificationMissedCount=0; RouteNotif
icationSequenceNumber=0}
ID : 52C1522F-C57A-4CB3-933B-F644BE4B7858
IPv6 : False
LayeredOn : C60CF73B-403D-4996-9876-42F0D61E90F1
MacPools : {@{EndMacAddress=00-15-5D-BA-8F-FF; StartMacAddress=00-15-5D-BA-80-00}}
MaxConcurrentEndpoints : 1
Name : nat
NatName : ICSAA72ADAD-A9D3-4425-8CCF-95F9AD9A922A
Policies : {}
Resources : @{AdditionalParams=; AllocationOrder=2; Allocators=System.Object[]; Health=; ID=FA7B5842-9A55-49F9-A483-1678F853A1B0; Por
tOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0; parentId=898003BD-B5EC-46B4-994F-CD1EA83759E3}
State : 1
Subnets : {@{AdditionalParams=; AddressPrefix=172.30.128.0/20; GatewayAddress=172.30.128.1; Health=; ID=6452007A-0D7B-4FE9-840A-E0A
1AEB73060; Policies=System.Object[]; State=0}}
TotalEndpoints : 1
Type : nat
Version : 38654705666
ActivityId : 5D96BBF3-E832-4E7A-9CAE-86E1EAE03934
AdditionalParams :
CurrentEndpointCount : 0
DNSServerCompartment : 5
DrMacAddress : 00-15-5D-9E-0F-93
Extensions : {@{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A; IsEnabled=False; Name=Microsoft Windows 筛选平台}, @{Id=E9B59CFA-2BE1-4B21-82
8F-B6FBDBDDC017; IsEnabled=True; Name=Microsoft Azure VFP Switch Extension}, @{Id=EA24CD6C-D17A-4348-9190-09F0D5BE83DD; I
sEnabled=True; Name=Microsoft NDIS 捕获}}
Flags : 0
Health : @{LastErrorCode=0; LastUpdateTime=132337298085630287}
ID : 0306A1CA-C7CF-49CB-870E-9C7AA94CFD3B
IPv6 : False
LayeredOn : BD7F3316-32A9-4380-9BA4-912B26BB57AB
MacPools : {@{EndMacAddress=00-15-5D-C7-5F-FF; StartMacAddress=00-15-5D-C7-50-00}}
ManagementIP : 192.168.8.126
MaxConcurrentEndpoints : 0
Name : External
Policies : {}
Resources : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; Health=; ID=5D96BBF3-E832-4E7A-9CAE-86E1EAE03934; Por
tOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0; parentId=74E57E4D-DD5A-4C31-B240-32233BD80A94}
State : 1
Subnets : {@{AdditionalParams=; AddressPrefix=192.168.255.0/30; GatewayAddress=192.168.255.1; Health=; ID=076EE1BC-189B-4C9C-BAF8-7
7F81F8DE90C; ObjectType=5; Policies=System.Object[]; State=0}}
TotalEndpoints : 0
Type : overlay
Version : 38654705666
ActivityId : BC1E4F7D-B057-44C5-ABA8-EE2D0D815642
AdditionalParams :
CurrentEndpointCount : 1
Extensions : {@{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A; IsEnabled=False; Name=Microsoft Windows 筛选平台}, @{Id=E9B59CFA-2BE1-4B21-82
8F-B6FBDBDDC017; IsEnabled=False; Name=Microsoft Azure VFP Switch Extension}, @{Id=EA24CD6C-D17A-4348-9190-09F0D5BE83DD;
IsEnabled=True; Name=Microsoft NDIS 捕获}}
Flags : 0
Health : @{AddressNotificationMissedCount=0; AddressNotificationSequenceNumber=0; InterfaceNotificationMissedCount=0; InterfaceNot
ificationSequenceNumber=0; LastErrorCode=0; LastUpdateTime=132337295318605785; RouteNotificationMissedCount=0; RouteNotif
icationSequenceNumber=0}
ID : 8327DCCF-F3AA-48B1-9A4B-F40D80219066
IPv6 : False
LayeredOn : C60CF73B-403D-4996-9876-42F0D61E90F1
MacPools : {@{EndMacAddress=00-15-5D-45-6F-FF; StartMacAddress=00-15-5D-45-60-00}}
MaxConcurrentEndpoints : 1
Name : 200866b762a74f6b539c310d17183071e94ea73c26f5cb25abbc8663898f5270
NatName : ICS1DF4177E-969B-4378-B8E2-8BF18DB1AF8A
Policies : {}
Resources : @{AdditionalParams=; AllocationOrder=2; Allocators=System.Object[]; Health=; ID=BC1E4F7D-B057-44C5-ABA8-EE2D0D815642; Por
tOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0; parentId=898003BD-B5EC-46B4-994F-CD1EA83759E3}
State : 1
Subnets : {@{AdditionalParams=; AddressPrefix=172.19.208.0/20; GatewayAddress=172.19.208.1; Health=; ID=14173728-B453-42C2-9B95-DE2
98953648E; Policies=System.Object[]; State=0}}
TotalEndpoints : 2
Type : nat
Version : 38654705666
ActivityId : 6F39C252-5915-484A-B5AC-B768F97062A2
AdditionalParams :
CurrentEndpointCount : 1
DNSServerCompartment : 6
DrMacAddress : 00-15-5D-9E-0F-93
Extensions : {@{Id=E7C3B2F0-F3C5-48DF-AF2B-10FED6D72E7A; IsEnabled=False; Name=Microsoft Windows 筛选平台}, @{Id=E9B59CFA-2BE1-4B21-82
8F-B6FBDBDDC017; IsEnabled=True; Name=Microsoft Azure VFP Switch Extension}, @{Id=EA24CD6C-D17A-4348-9190-09F0D5BE83DD; I
sEnabled=True; Name=Microsoft NDIS 捕获}}
Flags : 0
Health : @{LastErrorCode=0; LastUpdateTime=132337298346301336}
ID : B25F6D1F-4CFC-4E8C-9AC4-31C8B91856BA
IPv6 : False
LayeredOn : BD7F3316-32A9-4380-9BA4-912B26BB57AB
MacPools : {@{EndMacAddress=00-15-5D-8F-BF-FF; StartMacAddress=00-15-5D-8F-B0-00}}
ManagementIP : 192.168.8.126
MaxConcurrentEndpoints : 1
Name : vxlan0
Policies : {@{DestinationPrefix=10.244.0.0/24; DistributedRouterMacAddress=ae:d0:33:29:5c:a9; IsolationId=4096; ProviderAddress=192.
168.8.66; Type=RemoteSubnetRoute}}
Resources : @{AdditionalParams=; AllocationOrder=1; Allocators=System.Object[]; Health=; ID=6F39C252-5915-484A-B5AC-B768F97062A2; Por
tOperationTime=0; State=1; SwitchOperationTime=0; VfpOperationTime=0; parentId=74E57E4D-DD5A-4C31-B240-32233BD80A94}
State : 1
Subnets : {@{AdditionalParams=; AddressPrefix=10.244.2.0/24; GatewayAddress=10.244.2.1; Health=; ID=99819572-090F-439B-A3B7-532CD9E
14B7C; ObjectType=5; Policies=System.Object[]; State=0}}
TotalEndpoints : 2
Type : Overlay
Version : 38654705666
As part of installation for environment variables to take effect it's advised to start a new powershell session to perform JOIN. Issue is that if you are in servercore console you can not create a new session easily. The only option is either log off or reboot. Can description be modified to note that logoff is required for server core installs.
kubernetes version : v1.18.1
windows server 2019 10.0.17763.107
curl.exe -LO https://github.com/kubernetes-sigs/sig-windows-tools/releases/latest/download/PrepareNode.ps1
.\PrepareNode.ps1 -KubernetesVersion v1.18.1
Log file created at: 2020/04/28 21:54:37
Running on machine: w1
Binary: Built with gc go1.13.9 for windows/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
F0428 21:54:37.069887 1624 server.go:254] unable to load client CA file C:\var\lib\kubelet\etc\kubernetes\pki\ca.crt: open C:\var\lib\kubelet\etc\kubernetes\pki\ca.crt: The system cannot find the path specified.
goroutine 1 [running]:
k8s.io/kubernetes/vendor/k8s.io/klog.stacks(0xc000006001, 0xc000528000, 0x94b, 0x2710)
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:875 +0xbf
k8s.io/kubernetes/vendor/k8s.io/klog.(*loggingT).output(0x6c140a0, 0xc000000003, 0xc0009562a0, 0x6a8af81, 0x9, 0xfe, 0x0)
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:829 +0x188
k8s.io/kubernetes/vendor/k8s.io/klog.(*loggingT).printDepth(0x6c140a0, 0x3, 0x1, 0xc0008f7c60, 0x1, 0x1)
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:698 +0x130
k8s.io/kubernetes/vendor/k8s.io/klog.(*loggingT).print(...)
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:689
k8s.io/kubernetes/vendor/k8s.io/klog.Fatal(...)
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:1256
k8s.io/kubernetes/cmd/kubelet/app.NewKubeletCommand.func1(0xc0008fd680, 0xc000044210, 0x10, 0x1f)
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/app/server.go:254 +0x439
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0xc0008fd680, 0xc000044210, 0x10, 0x1f, 0xc0008fd680, 0xc000044210)
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:830 +0x2b1
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc0008fd680, 0x160a003c2d426e20, 0x6c13c00, 0xc000069f50)
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:914 +0x302
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(...)
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:864
main.main()
_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubelet/kubelet.go:41 +0xd4
goroutine 6 [chan receive]:
k8s.io/kubernetes/vendor/k8s.io/klog.(*loggingT).flushDaemon(0x6c140a0)
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:1010 +0x92
created by k8s.io/kubernetes/vendor/k8s.io/klog.init.0
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:411 +0xdd
goroutine 36 [sleep]:
runtime.goparkunlock(...)
/usr/local/go/src/runtime/proc.go:310
time.Sleep(0x3b9aca00)
/usr/local/go/src/runtime/time.go:105 +0x16c
k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1.(*metricsRecorder).run(0xc0001c2ea0)
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1/metrics_recorder.go:87 +0x46
created by k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1.newMetricsRecorder
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/scheduler/framework/v1alpha1/metrics_recorder.go:59 +0x10b
goroutine 59 [syscall]:
os/signal.signal_recv(0x461e66)
/usr/local/go/src/runtime/sigqueue.go:147 +0xa3
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:23 +0x29
created by os/signal.init.0
/usr/local/go/src/os/signal/signal_unix.go:29 +0x48
goroutine 67 [select]:
k8s.io/kubernetes/vendor/go.opencensus.io/stats/view.(*worker).start(0xc0007509b0)
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/go.opencensus.io/stats/view/worker.go:154 +0x107
created by k8s.io/kubernetes/vendor/go.opencensus.io/stats/view.init.0
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/go.opencensus.io/stats/view/worker.go:32 +0x5e
goroutine 78 [semacquire]:
sync.runtime_SemacquireMutex(0x6c140bc, 0x30000, 0x1)
/usr/local/go/src/runtime/sema.go:71 +0x4e
sync.(*Mutex).lockSlow(0x6c140b8)
/usr/local/go/src/sync/mutex.go:138 +0x103
sync.(*Mutex).Lock(...)
/usr/local/go/src/sync/mutex.go:81
k8s.io/kubernetes/vendor/k8s.io/klog.(*loggingT).lockAndFlushAll(0x6c140a0)
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:1017 +0x8b
k8s.io/kubernetes/vendor/k8s.io/klog.Flush()
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/klog/klog.go:438 +0x34
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x42804c0)
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x65
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x42804c0, 0x4842400, 0xc0009697d0, 0xc000446201, 0xc00007e0c0)
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xaa
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x42804c0, 0x12a05f200, 0x0, 0xc0005ec901, 0xc00007e0c0)
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0xe9
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until(...)
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Forever(0x42804c0, 0x12a05f200)
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:81 +0x56
created by k8s.io/kubernetes/vendor/k8s.io/component-base/logs.InitLogs
/workspace/anago-v1.18.1-beta.0.38+49aac775931dd1/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/component-base/logs/logs.go:58 +0x91
Twice, if forgotten to check that the docker service was actually started before executing "Kubecluster.ps1 ... -join". This causes the script to fail and you have to exit PS and restart before you can run it again.
I propose adding Start-Service Docker to the -join script to ensure the service is running before proceeding.
It may also be good to add Set-Service -Name Docker -StartupType Automatic to -install. For some reason the Docker installation does not always set this properly.
There is a scenario in which kubectl port-forward will not behave as expected, which also leads to the failure of the following tests:
[sig-cli] Kubectl Port forwarding [k8s.io] With a server listening on localhost [k8s.io] that expects NO client request should support a client that connects, sends DATA, and disconnects
[sig-cli] Kubectl Port forwarding [k8s.io] With a server listening on 0.0.0.0 [k8s.io] that expects NO client request should support a client that connects, sends DATA, and disconnects
Steps to reproduce this issue:
kubectl create namespace ns2
kubectl create -f port_fw2.yaml # [1]
kubectl port-forward -n ns2 pod/pfpod 9992:80 &
nc 127.0.0.1 9992 # this will output a number of x characters, and will hang.
kubectl logs -n ns2 pf/pod -c portforwardtester
Looking through the logs, the server lingers (making sure that all the data has been sent) before it closes the socket. I've added timestamps to the server's output:
Accepted client connection
Shutting down connection at 2019-07-15 17:12:38.7661425 +0000 GMT m=+516.300379001. Lingering...
After Lingering... 2019-07-15 17:12:39.4661261 +0000 GMT m=+517.000363301
Done at 2019-07-15 17:12:39.9686087 +0000 GMT m=+517.502868501
[1] https://paste.ubuntu.com/p/bW4mpYKV8W/
This issue was first signaled here: kubernetes/kubernetes#78857 (comment)
During the installtion with the script, flannel gets installed and a net-conf.json is generated under C:\ProgramData\Kubernetes.
Example content of net-conf.json under C:\ProgramData\Kubernetes:
{
"Network": "10.244.0.0/16",
"Backend": {
"name": "vxlan0",
"type": "vxlan"
}
}
But this updated configuration doesn't get copied to the c:\etc\kube-flannel\ folder, which means that the flanneld service is using an outdated configuration.
Example content of net-conf.json under C:\etc\kube-flannel (which is the original from https://github.com/microsoft/SDN/blob/master/Kubernetes/flannel/overlay/net-conf.json):
{
"Network": "192.168.0.0/16",
"Backend": {
"name": "vxlan0",
"type": "vxlan"
}
}
this PR added scripts for 1.15, but we probably should have 1.16 too if we are targeting 1.16 for the alpha:
https://github.com/kubernetes-sigs/sig-windows-tools/tree/master/kubeadm/v1.15.0
/kind feature
/priority important-soon
Executing ````curl.exe -k https://kubernetes.default.svc.cluster.local``` inside a Windows Pod, but get the following msg:
Using Flannel VxLan mode to create a Cluster with one control plane (via RKE) and one Windows Node (via https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/)
Executing into the Windows Pod
curl.exe <other service> is finecurl.exe https://www.google.com is finecurl.exe -k https://kubernetes.default.svc.cluster.local, but get timeout error
kubectl version):Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-08T17:38:50Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.1", GitCommit:"7879fc12a63337efff607952a323df90cdc7a335", GitTreeState:"clean", BuildDate:"2020-04-08T17:30:47Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
OS (e.g: cat /etc/os-release):
linux
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
windows:
Microsoft Windows [Version 10.0.18363.778]
windows hnsdiag info(hnsdiag list all):
PS C:\k> hnsdiag list all
////////////////////////NETWORKS///////////////////////
Network : EF66674E-8B36-4745-8BC0-7CCC3BAC1A02
Name : 79c8bbba59f2ef3b2f581b7ba6f4d93a16ebecd603edb0f1acd63ae30661a084
Type : nat
Subnet Address : 172.29.0.0/20
Gateway : 172.29.0.1
Network : 38F835EE-2BF6-4049-90BB-7C7C6ED5BAA8
Name : External
Type : Overlay
Subnet Address : 192.168.255.0/30
Gateway : 192.168.255.1
Network : 7B930604-5989-4729-B347-F8714F78F5CF
Name : vxlan0
Type : Overlay
Subnet Address : 10.244.7.0/24
Gateway : 10.244.7.1
///////////////////////ENDPOINTS///////////////////////
Endpoint : 8aa1be26-0e12-4479-8980-c3ad87cde921
Name : Ethernet
IP Address : 172.29.0.9
Endpoint : 2b9b4bf6-1659-4b2b-9178-a68207b9027e
Name : Ethernet
IP Address : 10.244.7.2
Endpoint : 2d0ef2c4-a60a-4d72-8c9a-a85d5bb0aae7
Name : Ethernet
IP Address : 10.96.0.10
Endpoint : dfe2694a-a85e-4015-b359-c6a9f89c9c08
Name : Ethernet
IP Address : 10.244.1.2
Endpoint : 2ed69ef6-6b5b-4bb7-887a-061f40827a03
Name : Ethernet
IP Address : 10.244.2.2
Endpoint : 56fe7577-442f-47e3-a995-06d2da90ad85
Name : Ethernet
IP Address : 10.97.197.166
Endpoint : 1f03934f-a171-4f7d-ae3f-d8b6b037b27f
Name : Ethernet
IP Address : 10.98.65.21
Endpoint : 91b1c24b-64e8-4339-b354-6a66197fcc57
Name : Ethernet
IP Address : 10.244.2.4
Endpoint : d5323104-8846-4af0-996a-f8816e8ac305
Name : Ethernet
IP Address : 10.244.0.2
Endpoint : d462212f-269d-478f-ba4f-8f8bd7ff51dc
Name : Ethernet
IP Address : 10.104.83.184
Endpoint : 3370b368-0cdc-41d3-a0e4-c637545a5820
Name : Ethernet
IP Address : 10.244.0.3
Endpoint : 24ae2a94-85ed-40ad-9292-7286a2ea0fac
Name : Ethernet
IP Address : 10.96.0.1
Endpoint : ae114b18-7e51-405d-90d0-e90890a87eaa
Name : Ethernet
IP Address : 172.16.127.47
Endpoint : 351cb9fd-3f92-4b9f-9a25-5c04029f70e1
Name : Ethernet
IP Address : 10.103.167.223
Endpoint : 033d0ea8-8bdb-4e99-84ba-7dac6283a106
Name : Ethernet
IP Address : 10.104.136.200
Endpoint : 2734eeb6-9422-4f21-895e-da07ba010759
Name : 7c57cb693fd0be420b722213afeddd5df335bbe7127266392ea4db1fc9d7af33_vxlan0
IP Address : 10.244.7.9
Endpoint : 9af94e8c-a103-4c4c-a802-a5434d98e4e1
Name : 0ed02a2566eaeeeb1b7a86a779283c8a5a9eda22d898289c33daeb1fa54ddbc4_vxlan0
IP Address : 10.244.7.11
//////////////////////NAMESPACES///////////////////////
Namespace : 910f7d92-ba2d-4c3f-98ae-7c0ac590d2dc
/////////////////////LOADBALANCERS/////////////////////
Load Balancer : c1a0c5e8-793c-45cf-8e09-50fa059128e1
Virtual IPs : 10.98.65.21
Direct IP IDs : d5323104-8846-4af0-996a-f8816e8ac305
Load Balancer : a349bf1b-7be0-4cf8-85d9-3e1d0ce22af3
Virtual IPs : 10.104.83.184
Direct IP IDs : 3370b368-0cdc-41d3-a0e4-c637545a5820
Load Balancer : c69aa297-e6f7-434e-b598-200d93f18ed2
Virtual IPs : 10.104.136.200
Direct IP IDs : 2734eeb6-9422-4f21-895e-da07ba010759
Load Balancer : b7ac2840-e42b-4178-aa4f-512c603851fb
Virtual IPs : 10.98.65.21
Direct IP IDs : d5323104-8846-4af0-996a-f8816e8ac305
Load Balancer : dc33716e-c614-4287-9ff6-5d74074071d1
Direct IP IDs : d5323104-8846-4af0-996a-f8816e8ac305
Load Balancer : 5998f3ee-bddc-45b3-a924-553c3ed510b7
Virtual IPs : 10.98.65.21
Direct IP IDs : d5323104-8846-4af0-996a-f8816e8ac305
Load Balancer : a7fb46a2-c09e-48e2-9b7e-10473cbd8f3b
Direct IP IDs : d5323104-8846-4af0-996a-f8816e8ac305
Load Balancer : 24df5760-997b-4cfc-9412-dbe3e3219700
Direct IP IDs : d5323104-8846-4af0-996a-f8816e8ac305
Load Balancer : 835e8f12-eae9-4919-aadd-2a1fdc3558c9
Virtual IPs : 10.96.0.10
Direct IP IDs : dfe2694a-a85e-4015-b359-c6a9f89c9c08
2ed69ef6-6b5b-4bb7-887a-061f40827a03
Load Balancer : e8c37ba7-aa91-414f-b470-d4c6a8c1a08e
Virtual IPs : 10.96.0.10
Direct IP IDs : dfe2694a-a85e-4015-b359-c6a9f89c9c08
2ed69ef6-6b5b-4bb7-887a-061f40827a03
Load Balancer : 3cc63908-5116-4250-8a2e-bb7d0c229a38
Virtual IPs : 10.96.0.10
Direct IP IDs : dfe2694a-a85e-4015-b359-c6a9f89c9c08
2ed69ef6-6b5b-4bb7-887a-061f40827a03
Load Balancer : 9feefa67-1ec9-499a-985e-8075e83c4429
Virtual IPs : 10.97.197.166
Direct IP IDs : 91b1c24b-64e8-4339-b354-6a66197fcc57
Load Balancer : a677cce8-7b6c-465d-83bb-65ade7e15cab
Virtual IPs : 10.98.65.21
Direct IP IDs : d5323104-8846-4af0-996a-f8816e8ac305
Load Balancer : 0f9ac3c7-d834-4281-b441-23c874d28275
Direct IP IDs : d5323104-8846-4af0-996a-f8816e8ac305
Load Balancer : aaf5545b-61f7-41dd-b099-06d3e713dfed
Virtual IPs : 10.98.65.21
Direct IP IDs : d5323104-8846-4af0-996a-f8816e8ac305
Load Balancer : 2c65a77a-59d4-40e1-a9f2-2d00059e9417
Direct IP IDs : d5323104-8846-4af0-996a-f8816e8ac305
Load Balancer : f08c8801-7a7f-4817-a5f6-6d25cdfecab3
Virtual IPs : 10.96.0.1
Direct IP IDs : ae114b18-7e51-405d-90d0-e90890a87eaa
when workers are joining the cluster we use the admin.conf the "kubeadm init" created and the user copied under "~/.kube".
sig-windows-tools/kubeadm/KubeClusterHelper.psm1
Lines 24 to 29 in 78f258a
giving workers admin rights is a security risk.
in my opinion we cannot graduate the kubeadm/windows feature to beta, unless this is resolved.
flannel does not require full blown admin rights to operate and according to @benmoss the privileges that it needs are here:
https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml#L49-L76
a potential solution here once the control-plane is started using "kubeadm init" the command "kubeadm alpha kubeconfig user ..." can be used to generate a new scoped kubeconfig for a group (ORG) that only has priviliges for this cluster-role https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml#L49-L76
this possibly means applying a new clusterrolebinding too.
/area security
/kind design
/priority important-longterm
cc @michmike
the users are confused...:
#61
we have other similar tickets here.
AFAIK, this project (kubeadm + sig-windwos_tools) supports 1903 only.
the k8s.io website documentation needs to be clear what versions of Windows we are supporting:
/priority important-soon
/sig windows
/kind documentation
/assign @benmoss @marosset
For windows node with multiple interfaces or multuple IP addresses. It's needed to configure NodeIP used by kubelet in such case. Currently PrepareNode.ps1 script does not support configure the Node IP on kubelet. So I think we can add a param in PrepareNode.ps1 to support the config.
Kubelet receives "--node-ip" for the setting.
Kubelet param:
--node-ip string
IP address of the node. If set, kubelet will use this IP address for the node
with the release of Beta support for kubeadm we should clean the following folders/files:
kubeadm/v1.15.0/*
kubeadm/v1.16.0/*
kubeadm/KubeCluster.ps1
kubeadm/KubeClusterHelper.psm1
/priority important-longterm
/kind cleanup
/help
I've followed all the steps on this page but they don't seem to be up-to-date / correct.
The flannel and kube-proxy containers won't start on Windows.
The flannel container has the following error message:
Warning FailedCreatePodSandBox 3m (x12 over 3m) kubelet, win-tplojsb0jn5 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "kube-flannel-ds-windows-amd64-wc22r": Error response from daemon: network host not found
The kube-proxy container has the following error message:
Warning FailedCreatePodSandBox 10m kubelet, win-tplojsb0jn5 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "76e22248a4c48eac683fad6f12c5d16077a1a1d0e58ffc499e12260e0ef87b85" network for pod "kube-proxy-windows-j2chr": networkPlugin cni failed to set up pod "kube-proxy-windows-j2chr_kube-system" network: error while GETHNSNewtorkByName(flannel.4096): Network flannel.4096 not found
Kubernetes Version:
v1.18.2
Windows Version:
Windows Server 2019 with latest updates
If there's any other information I can provide, please let me know
Hi,
I'm struggling with adding a windows node to my cluster.
I'm following the procedure on https://kubernetes.io/docs/setup/production-environment/windows/user-guide-windows-nodes/
I'm running windows server 2019 (1809 build:17763.805) on a virtual machine , I plan on using vxlan ( and already patched my cluster accordingly).
Here's my Kubeclustervxlan.json
{
"Cri" : {
"Name" : "dockerd",
"Images" : {
"Pause" : "mcr.microsoft.com/k8s/core/pause:1.2.0",
"Nanoserver" : "mcr.microsoft.com/windows/nanoserver:1809",
"ServerCore" : "mcr.microsoft.com/windows/servercore:ltsc2019"
}
},
"Cni" : {
"Name" : "flannel",
"Source" : [{
"Name" : "flanneld",
"Url" : "https://github.com/coreos/flannel/releases/download/v0.11.0/flanneld.exe"
}
],
"Plugin" : {
"Name": "vxlan"
},
"InterfaceName" : "Ethernet"
},
"Kubernetes" : {
"Source" : {
"Release" : "1.16.3",
"Url" : "https://dl.k8s.io/v1.16.3/kubernetes-node-windows-amd64.tar.gz"
},
"ControlPlane" : {
"IpAddress" : "123.123.123.123",
"Username" : "user",
"KubeadmToken" : "wwwww.wwwwwwwwwww",
"KubeadmCAHash" : "sha256:wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww"
},
"KubeProxy" : {
"Gates" : "WinOverlay=true"
},
"Network" : {
"ServiceCidr" : "10.96.0.0/12",
"ClusterCidr" : "10.41.0.0/16"
}
},
"Install" : {
"Destination" : "D:\\Kubernetes"
}
}The install script does not seem to retrieve the needed info from the ethernet interface , but it does not prevent the install either :
PS D:\transit\sig-windows-tools-master\kubeadm> .\KubeCluster.ps1 -ConfigFile .\v1.16.0\Kubeclustervxlan.json -install
Downloaded [https://raw.githubusercontent.com/Microsoft/SDN/master/Kubernetes/windows/hns.psm1] => [D:\transit\sig-windows-tools-master\kubeadm\hns.psm1]
WARNING: The names of some imported commands from the module 'hns' include unapproved verbs that might make them less
discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose
parameter. For a list of approved verbs, type Get-Verb.
Get-NetIPAddress : No matching MSFT_NetIPAddress objects found by CIM query for instances of the
ROOT/StandardCimv2/MSFT_NetIPAddress class on the CIM server: SELECT * FROM MSFT_NetIPAddress WHERE ((InterfaceAlias
LIKE 'vEthernet (Ethernet)')) AND ((AddressFamily = 2)). Verify query parameters and retry.
At D:\transit\sig-windows-tools-master\kubeadm\KubeClusterHelper.psm1:409 char:13
+ return (Get-NetIPAddress -InterfaceAlias "$InterfaceName" -Addres ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (MSFT_NetIPAddress:String) [Get-NetIPAddress], CimJobException
+ FullyQualifiedErrorId : CmdletizationQuery_NotFound,Get-NetIPAddress
Get-NetIPAddress : No matching MSFT_NetIPAddress objects found by CIM query for instances of the
ROOT/StandardCimv2/MSFT_NetIPAddress class on the CIM server: SELECT * FROM MSFT_NetIPAddress WHERE ((InterfaceAlias
LIKE 'vEthernet (Ethernet)')) AND ((AddressFamily = 2)). Verify query parameters and retry.
At D:\transit\sig-windows-tools-master\kubeadm\KubeClusterHelper.psm1:462 char:14
+ ... $addr = (Get-NetIPAddress -InterfaceAlias "$InterfaceName" -Addres ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (MSFT_NetIPAddress:String) [Get-NetIPAddress], CimJobException
+ FullyQualifiedErrorId : CmdletizationQuery_NotFound,Get-NetIPAddress
ConvertTo-DecimalIP : Cannot bind argument to parameter 'IPAddress' because it is null.
At D:\transit\sig-windows-tools-master\kubeadm\KubeClusterHelper.psm1:465 char:40
+ $mgmtSubnet = (ConvertTo-DecimalIP $addr) -band (ConvertTo-Decima ...
+ ~~~~~
+ CategoryInfo : InvalidData: (:) [ConvertTo-DecimalIP], ParameterBindingValidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,ConvertTo-DecimalIP
ConvertTo-MaskLength : Cannot bind argument to parameter 'SubnetMask' because it is null.
At D:\transit\sig-windows-tools-master\kubeadm\KubeClusterHelper.psm1:467 char:48
+ return "$mgmtSubnet/$(ConvertTo-MaskLength $mask)"
+ ~~~~~
+ CategoryInfo : InvalidData: (:) [ConvertTo-MaskLength], ParameterBindingValidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,ConvertTo-MaskLength
Directory: D:\Kubernetes
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 12/12/2019 10:05 AM logs
############################################
User Input
Destination : D:\Kubernetes
Master : 123.123.123.123
InterfaceName : Ethernet
Cri : dockerd
Cni : flannel
NetworkPlugin : vxlan
Release : 1.16.3
MasterIp : 123.123.123.123
ManagementIp :
ManagementSubnet : 0.0.0.0/
############################################
[DownloadFile] File D:\Kubernetes/kubernetes-node-windows-amd64.tar.gz already exists.
x kubernetes/
x kubernetes/node/
x kubernetes/node/bin/
x kubernetes/node/bin/kubelet.exe
x kubernetes/node/bin/kubectl.exe
x kubernetes/node/bin/kubeadm.exe
x kubernetes/node/bin/kube-proxy.exe
x kubernetes/LICENSES
x kubernetes/kubernetes-src.tar.gz
Downloading CNI binaries for overlay to D:\Kubernetes\cni
d----- 12/12/2019 10:05 AM cni
Directory: D:\Kubernetes\cni
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 12/12/2019 10:05 AM config
Downloading Flannel binaries
[DownloadFile] File D:\Kubernetes\flanneld.exe already exists.
[DownloadFile] File D:\Kubernetes/cni-plugins-windows-amd64-v0.8.2.tgz already exists.
x ./
x ./flannel.exe
x ./win-overlay.exe
x ./win-bridge.exe
x ./host-local.exe
C:\Users\svc_cftint/.ssh/id_rsa.pub
Execute the below commands on the Linux control-plane node (123.123.123.123) to add this Windows node's public key to its authorized keys
touch ~/.ssh/authorized_keys
echo >> ~/.ssh/authorized_keys
Please close this shell and open a new one to join this node to the clusterThen the join part fails
PS D:\transit\sig-windows-tools-master\kubeadm> .\KubeCluster.ps1 -ConfigFile .\v1.16.0\Kubeclustervxlan.json -join
[DownloadFile] File D:\transit\sig-windows-tools-master\kubeadm\hns.psm1 already exists.
WARNING: The names of some imported commands from the module 'hns' include unapproved verbs that might make them less
discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose
parameter. For a list of approved verbs, type Get-Verb.
Get-NetIPAddress : No matching MSFT_NetIPAddress objects found by CIM query for instances of the
ROOT/StandardCimv2/MSFT_NetIPAddress class on the CIM server: SELECT * FROM MSFT_NetIPAddress WHERE ((InterfaceAlias
LIKE 'vEthernet (Ethernet)')) AND ((AddressFamily = 2)). Verify query parameters and retry.
At D:\transit\sig-windows-tools-master\kubeadm\KubeClusterHelper.psm1:409 char:13
+ return (Get-NetIPAddress -InterfaceAlias "$InterfaceName" -Addres ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (MSFT_NetIPAddress:String) [Get-NetIPAddress], CimJobException
+ FullyQualifiedErrorId : CmdletizationQuery_NotFound,Get-NetIPAddress
Get-NetIPAddress : No matching MSFT_NetIPAddress objects found by CIM query for instances of the
ROOT/StandardCimv2/MSFT_NetIPAddress class on the CIM server: SELECT * FROM MSFT_NetIPAddress WHERE ((InterfaceAlias
LIKE 'vEthernet (Ethernet)')) AND ((AddressFamily = 2)). Verify query parameters and retry.
At D:\transit\sig-windows-tools-master\kubeadm\KubeClusterHelper.psm1:462 char:14
+ ... $addr = (Get-NetIPAddress -InterfaceAlias "$InterfaceName" -Addres ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : ObjectNotFound: (MSFT_NetIPAddress:String) [Get-NetIPAddress], CimJobException
+ FullyQualifiedErrorId : CmdletizationQuery_NotFound,Get-NetIPAddress
ConvertTo-DecimalIP : Cannot bind argument to parameter 'IPAddress' because it is null.
At D:\transit\sig-windows-tools-master\kubeadm\KubeClusterHelper.psm1:465 char:40
+ $mgmtSubnet = (ConvertTo-DecimalIP $addr) -band (ConvertTo-Decima ...
+ ~~~~~
+ CategoryInfo : InvalidData: (:) [ConvertTo-DecimalIP], ParameterBindingValidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,ConvertTo-DecimalIP
ConvertTo-MaskLength : Cannot bind argument to parameter 'SubnetMask' because it is null.
At D:\transit\sig-windows-tools-master\kubeadm\KubeClusterHelper.psm1:467 char:48
+ return "$mgmtSubnet/$(ConvertTo-MaskLength $mask)"
+ ~~~~~
+ CategoryInfo : InvalidData: (:) [ConvertTo-MaskLength], ParameterBindingValidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,ConvertTo-MaskLength
############################################
User Input
Destination : D:\Kubernetes
Master : 123.123.123.123
InterfaceName : Ethernet
Cri : dockerd
Cni : flannel
NetworkPlugin : vxlan
Release : 1.16.3
MasterIp : 123.123.123.123
ManagementIp :
ManagementSubnet : 0.0.0.0/
############################################
Downloading Kubeconfig from 10.179.17.71:~/.kube/config to D:\Kubernetes\config
config 100% 5448 5.3KB/s 00:00
Trying to connect to the Kubernetes control-plane node
############################################
Able to connect to the control-plane node
Discovered the following
Cluster CIDR : 10.41.0.0/16
Service CIDR : 10.96.0.0/12
DNS ServiceIp : 10.96.0.10
############################################
InstallKubelet : Cannot bind argument to parameter 'NodeIp' because it is an empty string.
At D:\transit\sig-windows-tools-master\kubeadm\KubeCluster.ps1:331 char:17
+ -NodeIp $Global:ManagementIp -KubeletFeatureGates $KubeletFea ...
+ ~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidData: (:) [InstallKubelet], ParameterBindingValidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationErrorEmptyStringNotAllowed,InstallKubelet
InstallFlannelD : Cannot bind argument to parameter 'InterfaceIpAddress' because it is an empty string.
At D:\transit\sig-windows-tools-master\kubeadm\KubeClusterHelper.psm1:1275 char:78
+ ... annelD -Destination $Global:BaseDir -InterfaceIpAddress $ManagementIp
+ ~~~~~~~~~~~~~
+ CategoryInfo : InvalidData: (:) [InstallFlannelD], ParameterBindingValidationException
+ FullyQualifiedErrorId : ParameterArgumentValidationErrorEmptyStringNotAllowed,InstallFlannelD
Generated CNI Config [{
"cniVersion": "0.2.0",
"name": "vxlan0",
"type": "flannel",
"capabilities": {
"dns": true
},
"delegate": {
"type": "win-overlay",
"Policies": [
{
"Name": "EndpointPolicy",
"Value": {
"Type": "OutBoundNAT",
"ExceptionList": [
"10.41.0.0/16",
"10.96.0.0/12"
]
}
},
{
"Name": "EndpointPolicy",
"Value": {
"Type": "ROUTE",
"DestinationPrefix": "10.96.0.0/12",
"NeedEncap": true
}
}
]
}
}]
Generated net-conf Config [{
"Network": "10.41.0.0/16",
"Backend": {
"name": "vxlan0",
"type": "vxlan"
}
}]
FlannelD service not installed
At D:\transit\sig-windows-tools-master\kubeadm\KubeClusterHelper.psm1:363 char:9
+ throw "FlannelD service not installed"
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : OperationStopped: (FlannelD service not installed:String) [], RuntimeException
+ FullyQualifiedErrorId : FlannelD service not installedThe join fails to create the Kubelet service because it can get the ManagementIP adress.
I'm a bit confused by that part... which IP is it supposed to be? I'm do not see it in Get-NetIPAddress for interface Ethernet...
I tried hard code the IP address of that interface but still got error ( I really suck at powershell and windows :( )
Any idea would be very welcome !! Thanks !!!
Hello,
I have an issue when i launch the KubeCluster.ps1. I've the error message bellow. I don't understand what is the key mentionned by the error message.
Can you help me ?
Unable to connect to the control-plane node. Reason [Item has already been added. Key in dictionary: 'Kube' Key being added: 'Kube']
At D:\sig-windows-tools-master\kubeadm\KubeCluster.ps1:311 char:9
throw "Unable to connect to the control-plane node. Reason [$ ...>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- CategoryInfo : OperationStopped: (Unable to conne... added: 'Kube']:String) [], >RuntimeException
- FullyQualifiedErrorId : Unable to connect to the control-plane node. Reason [Item has already >been added. Key in dictionary: 'Kube' Key being added: 'Kube']
requirement for beta graduation.
/assign
/kind feature
/priority important-longterm
I have the following setup
Windows 2019-1909
Kubernetes 1.18.2
Control Plane: CentOS 7.7 with k8s 1.18.2 built with kubedm
CNI: flannel with vxlan, using the proper vxlan ID and UDP ports for Windows compatibility
I followed PrepareNode.ps1 script here to get the 1909 server ready, but had to build my own kube-proxy and kube-flannel windows images as those don't support 1909. I had to build the setup.exe from another system and just ADD it into the container as there isn't a golang:servercore1909 image to use as the build image.
I've followed the instructions at https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/ to get everything up and running. I can successfully get a pod containing a servercore 1909 image. When I exec into this POD, I can ping all the linux cluster node IPs just fine. Route tables look accurate
However, when I try to reach the service network, including coredns, my connections time out. So I can't do DNS lookups whatsoever.
I can reach outside my cluster fine as well (nslookup using our physical DNS server IP addresses)
The only thing that doesn't seem to be working is service network connectivity. the node does have a proper route to the service network, and I can see \etc\cni\net.d\10-flannel.conf have the correct ExceptionList for OutBoundNAT that covers both the service network and pod networks, and also have a ROUTE type endpointpolicy with destination set to the service network with NeedEncap: true
Windows server containers need to be version matched to the host OS version when using the default "process" isolation. It would be more convenient if "KubeCluster.ps1" would check the windows version and automatically use the matching tag for the "Pause", "NanoServer" and "ServerCore" containers. This would be less error prone for new users than the current method of specifying in a config file.
I propose making these settings in the JSON split between container id and tag, for each, with the tag empty by default. When the tag is empty, the script would automatically determine based [System.Environment]::OSVersion.Version and use the appropriate version.
I'm happy to put in the work and submit a PR. I just want to make sure this would be useful to more than just myself and a welcome addition before doing it.
Thoughts?
Does not support running on the 1909 version system
Related to #36
Since Flannel is the one making the first HNS network right now, the network reconfiguration that happens is breaking the Flannel processes' ability to connect to Kubernetes.
You see in the flannel pods logs regular startup logging and then a bunch of lines like:
reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:307: Failed to list *v1.Node: Get https://10.128.0.12:6443/api/v1/nodes?resourceVersion=0: http2: no cached connection was available
I think if we fix #36 by initializing HNS in the PrepareNode script then this might be solved too. The downside of that is that it couples the PrepareNode script to networking, we might need to add a flag to the script to specify whether we set this network up as an overlay or l2bridge network.
We disabled automatic reboots for our e2e clusters in #51, but we should make it so even if a machine reboots that kubelet / docker / wins all are set to automatically start so that a machine can be rebooted.
Windows nodes, can't join to network with Calico.
It seems, it is supported only flannel.As the join parameters is asking for flannel details.
And also,
After joining node, if I restart the machine-(windows node). It is not recognizing the node again.
It is not showing the updated status after the machine is restarted.
i get a error that the kube-flannel-ds-windows-amd64 is down
------------
node version:
PS C:\Windows\system32> docker info
Client:
Debug Mode: false
Plugins:
cluster: Manage Docker Enterprise clusters (Mirantis Inc., v1.6.0)
Server:
Containers: 17
Running: 2
Paused: 0
Stopped: 15
Images: 4
Server Version: 19.03.12
Storage Driver: windowsfilter
Windows:
Logging Driver: json-file
Plugins:
Volume: local
Network: ics internal l2bridge l2tunnel nat null overlay private transparent
Log: awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog
Swarm: inactive
Default Isolation: process
Kernel Version: 10.0 17763 (17763.1.amd64fre.rs5_release.180914-1434)
Operating System: Windows Server 2019 Datacenter Version 1809 (OS Build 17763.1397)
OSType: windows
Architecture: x86_64
CPUs: 4
Total Memory: 7.812GiB
Name: WH-SR-11-203
ID: JJ7L:4MQY:S7PK:CJ2E:CI33:2JV3:6XFS:2HGT:CYSG:DMYB:7EU7:ANI4
Docker Root Dir: C:\ProgramData\docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
-------------
[root@drbd-master-02 new]# kubectl logs kube-flannel-ds-windows-amd64-4l8mv -n kube-system
Directory: C:\host\etc\cni
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 8/26/2020 3:30 AM net.d
Directory: C:\host\etc
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 8/26/2020 3:30 AM kube-flannel
Directory: C:\host\opt\cni
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 8/26/2020 3:30 AM bin
Directory: C:\host\k
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 8/26/2020 10:37 AM flannel
Directory: C:\host\k\flannel\var\run\secrets\kubernetes.io
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 8/26/2020 10:36 AM serviceaccount
����:
ģ�顰hns���е�ijЩ�������������ư���δ���Ķ��ʣ���Щ���ʿ��ܵ�����Щ���������ױ����֡���Ҫ���Ҿ���δ���Ķ��ʵ�����
����ʹ�� Verbose �����ٴ����� Import-Module ����й����Ķ����б��������� Get-Verb��
Invoke-HnsRequest : @{Error=�Ҳ����������� ; ErrorCode=2151350278; Success=False}
����λ�� C:\k\flannel\hns.psm1:233 �ַ�: 16
+ ... return Invoke-HnsRequest -Method POST -Type networks -Data $Json ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Invoke-HNSRequest
FATA[2020-08-26T10:38:53+08:00] rpc error: code = Internal desc = could not create IP forward entry: The object already exists.
I0826 10:38:54.791066 5160 main.go:518] Determining IP address of default interface
E0826 10:38:55.012530 5160 main.go:204] Failed to find any valid interface to use: failed to get default interface: no index found for interface ""
Hi i am trying to setup kubernetes cluster on 1903 like in documentation:
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/
But i get the following error instead,
I use image: sigwindowstools/flannel:0.12.0
Please help, starting windows cluster is crucial for my company
Warning FailedCreatePodSandBox 4s (x4 over 7s) kubelet, winkube-w2 (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "kube-flannel-ds-windows-amd64-fsnkw": Error response from daemon: hcsshim::CreateComputeSystem 0910a72ab593c5658d88b4f268fede8d4858c309ec790f7fe23a8805bafe01f9: The container operating system does not match the host operating system.
before restart computer ,windows node run well , kubelet and kube-proxy not print error logs
Log file created at: 2020/05/17 17:03:16
Running on machine: WIN-PB8V19E08A7
Binary: Built with gc go1.13.9 for windows/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0517 17:03:16.539124 6948 server.go:754] kubelet needs to run as uid `0`. It is being run as -1
E0517 17:03:16.602165 6948 file_unsupported.go:28] Watching source file is unsupported in this build
E0517 17:03:16.972802 6948 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
E0517 17:03:17.142229 6948 kubelet.go:1845] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
E0517 17:03:17.143226 6948 processstarttime.go:43] Could not get process start time, could not read /proc: CreateFile /proc: The system cannot find the file specified.
E0517 17:03:17.362888 6948 remote_runtime.go:495] ListContainerStats with filter &ContainerStatsFilter{Id:,PodSandboxId:,LabelSelector:map[string]string{},} from runtime service failed: rpc error: code = Unknown desc = hcsshim::OpenComputeSystem 3e2ee0013bb64a6296ef53a11119144297c64c083a5b4f3a16eb3f8151fa6cdf: A virtual machine or container with the specified identifier does not exist.
E0517 17:03:17.362888 6948 eviction_manager.go:255] eviction manager: failed to get summary stats: failed to list pod stats: failed to list all container stats: rpc error: code = Unknown desc = hcsshim::OpenComputeSystem 3e2ee0013bb64a6296ef53a11119144297c64c083a5b4f3a16eb3f8151fa6cdf: A virtual machine or container with the specified identifier does not exist.
E0517 17:03:17.377879 6948 cni.go:364] Error adding kube-system_kube-proxy-windows-htp85/a3bfa027b8fe55173dcfff3a9cc47a6380665d6bb69e87fdb68ca91d2be6bd37 to network flannel/flannel.4096: error while GETHNSNewtorkByName(flannel.4096): Network flannel.4096 not found
E0517 17:03:17.377879 6948 cni_windows.go:59] error while adding to cni network: error while GETHNSNewtorkByName(flannel.4096): Network flannel.4096 not found
I0426 02:13:40.790133 6128 proxier.go:368] Adding new service port "default/kubernetes:https" at 10.128.0.1:443/TCP
I0426 02:13:40.790133 6128 proxier.go:368] Adding new service port "kube-system/kube-dns:dns" at 10.128.0.10:53/UDP
I0426 02:13:40.790133 6128 proxier.go:368] Adding new service port "kube-system/kube-dns:dns-tcp" at 10.128.0.10:53/TCP
I0426 02:13:40.790133 6128 proxier.go:368] Adding new service port "kube-system/kube-dns:metrics" at 10.128.0.10:9153/TCP
E0426 02:13:40.796100 6128 proxier.go:1053] Source Vip endpoint creation failed: hcnCreateEndpoint failed in Win32: An address provided is invalid or reserved. (0x803b002f) {"Success":false,"Error":"An address provided is invalid or reserved. ","ErrorCode":2151350319}
The --interface argument for flannel setup, which calls the New-HnsNetwork powershell function does not seem to be necessary.
Should remove this from the flannel config files. The main kubernetes, https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/ documentation should also change to remove the need to modify this argument.
[root@m1 ~]# kubectl get pods -o=wide -A
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-system coredns-66bff467f8-2s9kt 1/1 Running 1 7d2h 10.244.0.5 m1 <none> <none>
kube-system coredns-66bff467f8-rqqq2 1/1 Running 1 7d2h 10.244.0.4 m1 <none> <none>
kube-system etcd-m1 1/1 Running 3569 7d2h 10.7.1.201 m1 <none> <none>
kube-system etcd-m2 1/1 Running 7 7d2h 10.7.1.202 m2 <none> <none>
kube-system etcd-m3 1/1 Running 1 7d2h 10.7.1.203 m3 <none> <none>
kube-system kube-apiserver-m1 1/1 Running 4266 7d2h 10.7.1.201 m1 <none> <none>
kube-system kube-apiserver-m2 1/1 Running 8 7d2h 10.7.1.202 m2 <none> <none>
kube-system kube-apiserver-m3 1/1 Running 1 7d2h 10.7.1.203 m3 <none> <none>
kube-system kube-controller-manager-m1 1/1 Running 7 7d2h 10.7.1.201 m1 <none> <none>
kube-system kube-controller-manager-m2 1/1 Running 6 7d2h 10.7.1.202 m2 <none> <none>
kube-system kube-controller-manager-m3 0/1 CrashLoopBackOff 1 7d2h 10.7.1.203 m3 <none> <none>
kube-system kube-flannel-ds-amd64-2nhtf 0/1 Pending 0 27h <none> m3 <none> <none>
kube-system kube-flannel-ds-amd64-6bdw6 1/1 Running 0 27h 10.7.1.202 m2 <none> <none>
kube-system kube-flannel-ds-amd64-cpfmr 1/1 Running 0 27h 10.7.1.201 m1 <none> <none>
kube-system kube-flannel-ds-amd64-px8b2 1/1 Running 0 27h 10.7.1.211 n1 <none> <none>
kube-system kube-flannel-ds-amd64-v545q 1/1 Terminating 1 7d2h 10.7.1.203 m3 <none> <none>
kube-system kube-flannel-ds-windows-amd64-646kj 0/1 Pending 0 12m <none> w1 <none> <none>
kube-system kube-proxy-fs47v 1/1 Running 5 7d2h 10.7.1.202 m2 <none> <none>
kube-system kube-proxy-jnstv 1/1 Running 1 7d2h 10.7.1.203 m3 <none> <none>
kube-system kube-proxy-q4stn 1/1 Running 1 7d2h 10.7.1.211 n1 <none> <none>
kube-system kube-proxy-windows-xcrt2 0/1 Pending 0 12m <none> w1 <none> <none>
kube-system kube-proxy-zgl5q 1/1 Running 3 7d2h 10.7.1.201 m1 <none> <none>
kube-system kube-scheduler-m1 1/1 Running 5 7d2h 10.7.1.201 m1 <none> <none>
kube-system kube-scheduler-m2 1/1 Running 7 7d2h 10.7.1.202 m2 <none> <none>
kube-system kube-scheduler-m3 1/1 Running 1 7d2h 10.7.1.203 m3 <none> <none>
PS C:\> docker pull sigwindowstools/flannel Using default tag: latest
Error response from daemon: manifest for sigwindowstools/flannel:latest not found: manifest unknown: manifest unknown
Hitting the same issue as microsoft/hcsshim#299
HNS gets initialized right now in our flow when the Flannel pod runs. We need to explore options for manually re-adding a route to 169.254.169.254 ala this code after HNS initialization.
Options I can imagine:
kubernetes 1.18.1
docker 19.03.5
����:
ģ�顰hns���е�ijЩ�������������ư���δ���Ķ��ʣ���Щ���ʿ��ܵ�����Щ���������ױ����֡���Ҫ���Ҿ���δ���Ķ��ʵ�����
����ʹ�� Verbose �����ٴ����� Import-Module ����й����Ķ����б��������� Get-Verb��
Invoke-HnsRequest : @{Error=�Ҳ����������� ; ErrorCode=2151350278; Success=False}
����λ�� C:\k\flannel\hns.psm1:233 �ַ�: 16
+ ... return Invoke-HnsRequest -Method POST -Type networks -Data $Json ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Invoke-HNSRequest
FATA[2020-04-20T18:31:47+08:00] rpc error: code = Internal desc = could not create IP forward entry: The object already exists.
I0420 18:31:48.571762 2888 main.go:518] Determining IP address of default interface
E0420 18:31:48.942744 2888 main.go:204] Failed to find any valid interface to use: failed to get default interface: no index found for interface ""
Hello,
Followed instructions down to the t and windows 1909 host fails to join 1.18.0. I followed instructions from here (https://deploy-preview-19217--kubernetes-io-vnext-staging.netlify.com/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/)
Part of Kubelet log is below
Log file created at: 2020/03/26 23:54:37
Running on machine: winworker2
Binary: Built with gc go1.13.8 for windows/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0326 23:54:37.534257 4952 server.go:417] Version: v1.18.0
I0326 23:54:37.575309 4952 plugins.go:100] No cloud provider specified.
I0326 23:54:37.576312 4952 server.go:837] Client rotation is on, will bootstrap in background
I0326 23:54:37.607356 4952 certificate_store.go:130] Loading cert/key pair from "C:\\var\\lib\\kubelet\\pki\\kubelet-client-current.pem".
E0326 23:54:38.191095 4952 server.go:754] kubelet needs to run as uid `0`. It is being run as -1
W0326 23:54:38.193095 4952 server.go:762] setting OOM scores is unsupported in this build
I0326 23:54:38.197111 4952 client.go:92] Start docker client with request timeout=2m0s
W0326 23:54:38.248203 4952 docker_service.go:561] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
I0326 23:54:38.248203 4952 docker_service.go:238] Hairpin mode set to "hairpin-veth"
W0326 23:54:38.249191 4952 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
W0326 23:54:38.249191 4952 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
I0326 23:54:38.249191 4952 docker_service.go:253] Docker cri networking managed by cni
W0326 23:54:38.249191 4952 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
I0326 23:54:38.260220 4952 docker_service.go:258] Docker Info: &{ID:W3W5:QKTS:KLUJ:LPHI:YRBP:72X4:PKKI:HM4S:HMVO:5AYQ:XVMI:PV6H Containers:1 ContainersRunning:0 ContainersPaused:0 ContainersStopped:1 Images:4 Driver:windowsfilter DriverStatus:[[Windows ]] SystemStatus:[] Plugins:{Volume:[local] Network:[ics internal l2bridge l2tunnel nat null overlay private transparent] Authorization:[] Log:[awslogs etwlogs fluentd gcplogs gelf json-file local logentries splunk syslog]} MemoryLimit:false SwapLimit:false KernelMemory:false KernelMemoryTCP:false CPUCfsPeriod:false CPUCfsQuota:false CPUShares:false CPUSet:false PidsLimit:false IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6tables:true Debug:false NFd:-1 OomKillDisable:false NGoroutines:27 SystemTime:2020-03-26T23:54:38.2501948Z LoggingDriver:json-file CgroupDriver: NEventsListener:0 KernelVersion:10.0 18363 (18362.1.amd64fre.19h1_release.190318-1202) OperatingSystem:Windows Server Datacenter Version 1909 (OS Build 18363.720) OSType:windows Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:0xc000283500 NCPU:2 MemTotal:8589463552 GenericResources:[] DockerRootDir:C:\ProgramData\docker HTTPProxy: HTTPSProxy: NoProxy: Name:winworker2 Labels:[] ExperimentalBuild:false ServerVersion:19.03.5 ClusterStore: ClusterAdvertise: Runtimes:map[] DefaultRuntime: Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil> Warnings:[]} LiveRestoreEnabled:false Isolation:process InitBinary: ContainerdCommit:{ID: Expected:} RuncCommit:{ID: Expected:} InitCommit:{ID: Expected:} SecurityOptions:[] ProductLicense: Warnings:[]}
W0326 23:54:38.260220 4952 docker_service.go:263] No cgroup driver is set in Docker
W0326 23:54:38.260220 4952 docker_service.go:264] Falling back to use the default driver: "cgroupfs"
I0326 23:54:38.260220 4952 docker_service.go:271] Setting cgroupDriver to cgroupfs
W0326 23:54:38.260220 4952 docker_service.go:425] initialization error: unable to list values under registry key "SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Virtualization\\Containers\\CredentialSpecs": Access is denied.
I0326 23:54:38.292276 4952 remote_runtime.go:59] parsed scheme: ""
I0326 23:54:38.292276 4952 remote_runtime.go:59] scheme "" not registered, fallback to default scheme
I0326 23:54:38.294272 4952 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{//./pipe/dockershim <nil> 0 <nil>}] <nil> <nil>}
I0326 23:54:38.294272 4952 clientconn.go:933] ClientConn switching balancer to "pick_first"
I0326 23:54:38.295274 4952 remote_image.go:50] parsed scheme: ""
I0326 23:54:38.295274 4952 remote_image.go:50] scheme "" not registered, fallback to default scheme
I0326 23:54:38.295274 4952 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{//./pipe/dockershim <nil> 0 <nil>}] <nil> <nil>}
I0326 23:54:38.295274 4952 clientconn.go:933] ClientConn switching balancer to "pick_first"
I0326 23:54:38.296275 4952 kubelet.go:292] Adding pod path: C:\var\lib\kubelet\etc\kubernetes\manifests
E0326 23:54:38.298283 4952 file_unsupported.go:28] Watching source file is unsupported in this build
I0326 23:54:38.298283 4952 kubelet.go:317] Watching apiserver
W0326 23:54:38.302283 4952 iptables.go:207] Error checking iptables version, assuming version at least 1.4.11: executable file not found in %PATH%
E0326 23:54:38.303317 4952 file.go:98] Unable to read config path "C:\\var\\lib\\kubelet\\etc\\kubernetes\\manifests": path does not exist, ignoring
I0326 23:54:38.411178 4952 state_mem.go:36] [cpumanager] initializing new in-memory state store
I0326 23:54:38.411178 4952 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
E0326 23:54:38.547582 4952 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
For verbose messaging see aws.Config.CredentialsChainVerboseErrors
I0326 23:54:39.002133 4952 kuberuntime_manager.go:211] Container runtime docker initialized, version: 19.03.5, apiVersion: 1.40.0
I0326 23:54:39.014153 4952 server.go:1125] Started kubelet
I0326 23:54:39.014153 4952 server.go:145] Starting to listen on 0.0.0.0:10250
E0326 23:54:39.043200 4952 processstarttime.go:43] Could not get process start time, could not read /proc: CreateFile /proc: The system cannot find the file specified.
I0326 23:54:39.044210 4952 server.go:393] Adding debug handlers to kubelet server.
I0326 23:54:39.047207 4952 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
I0326 23:54:39.048217 4952 status_manager.go:158] Starting to sync pod status with apiserver
I0326 23:54:39.048217 4952 kubelet.go:1821] Starting kubelet main sync loop.
E0326 23:54:39.048217 4952 kubelet.go:1845] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
I0326 23:54:39.050218 4952 volume_manager.go:265] Starting Kubelet Volume Manager
I0326 23:54:39.055224 4952 desired_state_of_world_populator.go:139] Desired state populator starts to run
E0326 23:54:39.099298 4952 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
I0326 23:54:39.102297 4952 plugin_manager.go:114] Starting Kubelet Plugin Manager
E0326 23:54:39.107305 4952 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
W0326 23:54:39.149536 4952 pod_container_deletor.go:77] Container "281c1960b981f158c37965767c3e311e8b868534ccc6ea1e87e4a7a6867af5d1" not found in pod's containers
E0326 23:54:39.150618 4952 pod_workers.go:191] Error syncing pod 1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe ("kube-proxy-windows-slwj8_kube-system(1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe)"), skipping: network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
I0326 23:54:39.152532 4952 kuberuntime_manager.go:978] updating runtime config through cri with podcidr 10.244.1.0/24
I0326 23:54:39.152532 4952 kubelet_node_status.go:294] Setting node annotation to enable volume controller attach/detach
I0326 23:54:39.154539 4952 docker_service.go:353] docker cri received runtime config &RuntimeConfig{NetworkConfig:&NetworkConfig{PodCidr:10.244.1.0/24,},}
I0326 23:54:39.202827 4952 kubelet_network.go:77] Setting Pod CIDR: -> 10.244.1.0/24
I0326 23:54:39.204823 4952 kubelet_node_status.go:70] Attempting to register node winworker2
E0326 23:54:39.206831 4952 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
I0326 23:54:39.267925 4952 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "wins" (UniqueName: "kubernetes.io/host-path/d16c4ffb-a4dd-43ab-b08a-c7efaf8313dc-wins") pod "kube-flannel-ds-windows-amd64-47qlx" (UID: "d16c4ffb-a4dd-43ab-b08a-c7efaf8313dc")
I0326 23:54:39.267925 4952 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe-kube-proxy") pod "kube-proxy-windows-slwj8" (UID: "1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe")
I0326 23:54:39.267925 4952 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "host" (UniqueName: "kubernetes.io/host-path/1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe-host") pod "kube-proxy-windows-slwj8" (UID: "1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe")
I0326 23:54:39.267925 4952 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "wins" (UniqueName: "kubernetes.io/host-path/1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe-wins") pod "kube-proxy-windows-slwj8" (UID: "1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe")
I0326 23:54:39.267925 4952 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy-token-x4l8l" (UniqueName: "kubernetes.io/secret/1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe-kube-proxy-token-x4l8l") pod "kube-proxy-windows-slwj8" (UID: "1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe")
I0326 23:54:39.267925 4952 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "host" (UniqueName: "kubernetes.io/host-path/d16c4ffb-a4dd-43ab-b08a-c7efaf8313dc-host") pod "kube-flannel-ds-windows-amd64-47qlx" (UID: "d16c4ffb-a4dd-43ab-b08a-c7efaf8313dc")
I0326 23:54:39.267925 4952 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "cni" (UniqueName: "kubernetes.io/host-path/d16c4ffb-a4dd-43ab-b08a-c7efaf8313dc-cni") pod "kube-flannel-ds-windows-amd64-47qlx" (UID: "d16c4ffb-a4dd-43ab-b08a-c7efaf8313dc")
I0326 23:54:39.267925 4952 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/d16c4ffb-a4dd-43ab-b08a-c7efaf8313dc-kube-proxy") pod "kube-flannel-ds-windows-amd64-47qlx" (UID: "d16c4ffb-a4dd-43ab-b08a-c7efaf8313dc")
I0326 23:54:39.267925 4952 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "kubeadm-config" (UniqueName: "kubernetes.io/configmap/d16c4ffb-a4dd-43ab-b08a-c7efaf8313dc-kubeadm-config") pod "kube-flannel-ds-windows-amd64-47qlx" (UID: "d16c4ffb-a4dd-43ab-b08a-c7efaf8313dc")
I0326 23:54:39.267925 4952 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "flannel-token-f4ctv" (UniqueName: "kubernetes.io/secret/d16c4ffb-a4dd-43ab-b08a-c7efaf8313dc-flannel-token-f4ctv") pod "kube-flannel-ds-windows-amd64-47qlx" (UID: "d16c4ffb-a4dd-43ab-b08a-c7efaf8313dc")
I0326 23:54:39.267925 4952 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "kube-proxy-windows" (UniqueName: "kubernetes.io/configmap/1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe-kube-proxy-windows") pod "kube-proxy-windows-slwj8" (UID: "1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe")
I0326 23:54:39.267925 4952 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "flannel-cfg" (UniqueName: "kubernetes.io/configmap/d16c4ffb-a4dd-43ab-b08a-c7efaf8313dc-flannel-cfg") pod "kube-flannel-ds-windows-amd64-47qlx" (UID: "d16c4ffb-a4dd-43ab-b08a-c7efaf8313dc")
I0326 23:54:39.267925 4952 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "flannel-windows-cfg" (UniqueName: "kubernetes.io/configmap/d16c4ffb-a4dd-43ab-b08a-c7efaf8313dc-flannel-windows-cfg") pod "kube-flannel-ds-windows-amd64-47qlx" (UID: "d16c4ffb-a4dd-43ab-b08a-c7efaf8313dc")
I0326 23:54:39.268928 4952 reconciler.go:157] Reconciler: start to sync state
W0326 23:54:39.453529 4952 docker_sandbox.go:238] Both sandbox container and checkpoint for id "281c1960b981f158c37965767c3e311e8b868534ccc6ea1e87e4a7a6867af5d1" could not be found. Proceed without further sandbox information.
E0326 23:54:39.454531 4952 remote_runtime.go:128] StopPodSandbox "281c1960b981f158c37965767c3e311e8b868534ccc6ea1e87e4a7a6867af5d1" from runtime service failed: rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod "_" network: cni config uninitialized
E0326 23:54:39.454531 4952 kuberuntime_manager.go:895] Failed to stop sandbox {"docker" "281c1960b981f158c37965767c3e311e8b868534ccc6ea1e87e4a7a6867af5d1"}
E0326 23:54:39.454531 4952 kuberuntime_manager.go:674] killPodWithSyncResult failed: failed to "KillPodSandbox" for "d16c4ffb-a4dd-43ab-b08a-c7efaf8313dc" with KillPodSandboxError: "rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"_\" network: cni config uninitialized"
E0326 23:54:39.454531 4952 pod_workers.go:191] Error syncing pod d16c4ffb-a4dd-43ab-b08a-c7efaf8313dc ("kube-flannel-ds-windows-amd64-47qlx_kube-system(d16c4ffb-a4dd-43ab-b08a-c7efaf8313dc)"), skipping: failed to "KillPodSandbox" for "d16c4ffb-a4dd-43ab-b08a-c7efaf8313dc" with KillPodSandboxError: "rpc error: code = Unknown desc = networkPlugin cni failed to teardown pod \"_\" network: cni config uninitialized"
I0326 23:54:39.831065 4952 kubelet_node_status.go:112] Node winworker2 was previously registered
I0326 23:54:39.831065 4952 kubelet_node_status.go:73] Successfully registered node winworker2
I0326 23:54:39.847092 4952 setters.go:559] Node became not ready: {Type:Ready Status:False LastHeartbeatTime:2020-03-26 23:54:39.847092 +0000 GMT m=+3.633765901 LastTransitionTime:2020-03-26 23:54:39.847092 +0000 GMT m=+3.633765901 Reason:KubeletNotReady Message:runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized}
E0326 23:54:41.055823 4952 pod_workers.go:191] Error syncing pod 1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe ("kube-proxy-windows-slwj8_kube-system(1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe)"), skipping: network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
E0326 23:54:43.055362 4952 pod_workers.go:191] Error syncing pod 1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe ("kube-proxy-windows-slwj8_kube-system(1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe)"), skipping: network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
W0326 23:54:43.256908 4952 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
E0326 23:54:44.116192 4952 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
E0326 23:54:45.063469 4952 pod_workers.go:191] Error syncing pod 1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe ("kube-proxy-windows-slwj8_kube-system(1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe)"), skipping: network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
E0326 23:54:47.060186 4952 pod_workers.go:191] Error syncing pod 1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe ("kube-proxy-windows-slwj8_kube-system(1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe)"), skipping: network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
W0326 23:54:48.265722 4952 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
E0326 23:54:49.064425 4952 pod_workers.go:191] Error syncing pod 1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe ("kube-proxy-windows-slwj8_kube-system(1dbeb45a-eea4-49ff-87fc-1b0b2279fdfe)"), skipping: network is not ready: runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
E0326 23:54:49.122828 4952 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Hi, We have a 2 linux, 1 windows node cluster setup. The linux nodes work and the windows node shows ready from a kubernetes cluster perspective. However when trying to setup the nginx-ingress controller, the windows node is trying to pull the linux image
Normal Scheduled 11m default-scheduler Successfully assigned nginx-ingress/nginx-ingress-tr77s to aabrw-kuber03
Normal Pulling 9m30s (x4 over 10m) kubelet, aabrw-kuber03 Pulling image "nginx/nginx-ingress:1.8.1"
Warning Failed 9m29s (x4 over 10m) kubelet, aabrw-kuber03 Error: ErrImagePull
Warning Failed 52s (x41 over 10m) kubelet, aabrw-kuber03 Error: ImagePullBackOff
the other nodes (linux) are running nginx-ingress.
[[email protected]@aabrl-kuber01 daemon-set]$ kubectl get pods --namespace nginx-ingress
NAME READY STATUS RESTARTS AGE
nginx-ingress-94d765bfd-64p6q 1/1 Running 2 6h30m
nginx-ingress-gs4mt 1/1 Running 2 27m
nginx-ingress-lbxp4 1/1 Running 0 42m
nginx-ingress-tr77s 0/1 ImagePullBackOff 0 11m
How do we force the windows node to pull the right image?
Hello
i installed a Mixed Kubernetes 1.17.6 cluster with Flannel host-gw (l2bridge) by using flannel-host-gw.yml file.
It worked fine until Windows node reboot.
After reboot, flannel windows pod doesn't start.
It stays in ContainerCreating state.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 58m default-scheduler Successfully assigned kube-system/kube-flannel-ds-windows-amd64-hbv8t to kub2win01
Warning FailedCreatePodSandBox 13m (x2459 over 58m) kubelet, kub2win01 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "kube-flannel-ds-windows-amd64-hbv8t": Error response from daemon: network host not found
Normal SandboxChanged 3m9s (x3001 over 58m) kubelet, kub2win01 Pod sandbox changed, it will be killed and re-created.
kube-proxy windows daemonset pod has been successfully restarted after reboot
Thank you
Regards
Stéphane
After following the guid for join window node ,I ran the prepare node script and then joined the kubeadm join , I made it works on my cluster. But the windows nodes is always not ready.
According to the results of running kubectl describe node nodename, network plugin is not ready: cni config uninitialized.
The output of kubectl describe pod kube-flannel-ds-windows-amd64-n2dzk -n kube-system is shown as below.
Name: kube-flannel-ds-windows-amd64-n2dzk
Namespace: kube-system
Priority: 0
Node: win-2bvbjuavi8h/10.0.75.1
Start Time: Mon, 30 Mar 2020 02:57:33 +0800
Labels: app=flannel
controller-revision-hash=5c69bc498d
pod-template-generation=1
tier=node
Annotations: <none>
Status: Pending
IP: 10.0.75.1
IPs:
IP: 10.0.75.1
Controlled By: DaemonSet/kube-flannel-ds-windows-amd64
Containers:
kube-flannel:
Container ID:
Image: sigwindowstools/flannel:0.12.0
Image ID:
Port: <none>
Host Port: <none>
Command:
powershell
Args:
-file
/etc/kube-flannel-windows/run.ps1
State: Waiting
Reason: CreateContainerError
Ready: False
Restart Count: 0
Environment:
POD_NAME: kube-flannel-ds-windows-amd64-n2dzk (v1:metadata.name)
POD_NAMESPACE: kube-system (v1:metadata.namespace)
Mounts:
/etc/cni/net.d from cni (rw)
/etc/kube-flannel-windows/ from flannel-windows-cfg (rw)
/etc/kube-flannel/ from flannel-cfg (rw)
/etc/kubeadm-config/ from kubeadm-config (rw)
/host from host (rw)
/kube-proxy from kube-proxy (rw)
/var/run/secrets/kubernetes.io/serviceaccount from flannel-token-jrdz2 (ro)
\\.\pipe\rancher_wins from wins (rw)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
opt:
Type: HostPath (bare host directory volume)
Path: /opt
HostPathType:
host:
Type: HostPath (bare host directory volume)
Path: /
HostPathType:
cni:
Type: HostPath (bare host directory volume)
Path: /etc
HostPathType:
flannel-cfg:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kube-flannel-cfg
Optional: false
flannel-windows-cfg:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kube-flannel-windows-cfg
Optional: false
kube-proxy:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kube-proxy
Optional: false
kubeadm-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kubeadm-config
Optional: false
wins:
Type: HostPath (bare host directory volume)
Path: \\.\pipe\rancher_wins
HostPathType:
flannel-token-jrdz2:
Type: Secret (a volume populated by a Secret)
SecretName: flannel-token-jrdz2
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: :NoSchedule
node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/network-unavailable:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/pid-pressure:NoSchedule
node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unschedulable:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreatePodSandBox 18m (x3 over 18m) kubelet, win-2bvbjuavi8h Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create a sandbox for pod "kube-flannel-ds-windows-amd64-n2dzk": Error response from daemon: Conflict. The container name "/k8s_POD_kube-flannel-ds-windows-amd64-n2dzk_kube-system_4b55d54c-8bc6-4414-a497-00d24b7519f7_0" is already in use by container "1d4fe16fd74e6d84a04173b9e624591f44c444a9956b9d332deae1108386dd1c". You have to remove (or rename) that container to be able to reuse that name.
Warning Failed 17m (x5 over 21m) kubelet, win-2bvbjuavi8h Error: Error response from daemon: Conflict. The container name "/k8s_kube-flannel_kube-flannel-ds-windows-amd64-n2dzk_kube-system_4b55d54c-8bc6-4414-a497-00d24b7519f7_0" is already in use by container "fda72631b4e0634fbd31068b97a06dbdd8cbe2b13f97f7ce2375e3956613ffdb". You have to remove (or rename) that container to be able to reuse that name.
Warning FailedCreatePodSandBox 17m kubelet, win-2bvbjuavi8h Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create a sandbox for pod "kube-flannel-ds-windows-amd64-n2dzk": Error response from daemon: Conflict. The container name "/k8s_POD_kube-flannel-ds-windows-amd64-n2dzk_kube-system_4b55d54c-8bc6-4414-a497-00d24b7519f7_0" is already in use by container "e0c3e13c5346d9e0b3d9f92025dbbe47c2cde4c13571e3cbf39cdbb8aa2b03c7". You have to remove (or rename) that container to be able to reuse that name.
Warning FailedCreatePodSandBox 2m49s (x3 over 3m16s) kubelet, win-2bvbjuavi8h Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create a sandbox for pod "kube-flannel-ds-windows-amd64-n2dzk": Error response from daemon: Conflict. The container name "/k8s_POD_kube-flannel-ds-windows-amd64-n2dzk_kube-system_4b55d54c-8bc6-4414-a497-00d24b7519f7_0" is already in use by container "7d0f874fadb6fb9a150d84c9e645a590d279b9d09022702e8a2c3c9f4f4c7bb6". You have to remove (or rename) that container to be able to reuse that name.
there are always events about container name conflict, but the output of docker ps -a is nothing , no container is running or was running on the windows node.
The kubernetes version used in my cluster is v1.18.0, the version of docker on the windows node is 18.09.2, but this situation also appears when I used v1.17.4 kubernetes.
I also have the images needed on my windows node,this is the images.
PS C:\Users\Administrator> docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
sigwindowstools/kube-proxy v1.18.0 68845b910f08 20 hours ago 5.03GB
sigwindowstools/flannel 0.12.0 9499a92cb176 5 days ago 5.06GB
mcr.microsoft.com/windows/servercore ltsc2019 80e84fd280e2 3 weeks ago 4.94GB
mcr.microsoft.com/windows/nanoserver 1809 39a578dbf5b5 3 weeks ago 251MB
mcr.microsoft.com/oss/kubernetes/pause 1.3.0 e2b9b3d368da 7 weeks ago 256MB
mcr.microsoft.com/k8s/core/pause 1.2.0 a74290a8271a 9 months ago 253MB
I have been stucked with this problem for a long time. I need help for windows node joining.
I am currently running my cluster with a virtual IP (172.20.2.1) behind an HAProxy with three master modes (172.20.1.85, 172.20.1.88, 172.20.1.90).
When running the join script, the windows node fails to join the cluster.
This appears to be due to GetAPIServerEndpoint returning the full master list, with an appended port (172.20.1.85 172.20.1.88 172.20.1.90:6443) rather than either choosing one of the IPs it finds, or utilizing the IP provided in kubeclustervxlan.json
For some reason by Windows node fails to use networking properly. It has been working fine for last 2 weeks. I reinstalled K8 on windows node as well but still fails. FlannelD service is in Stopped state and failed to start. I'm running 1.17.4.
I have no option to install via kubadm due to this #61 (comment)
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "eed0f7a03c72d9292e9683d3f800f07c497a37f6d07e313c5080296dfc67cbdd" network for pod "win-webserver-fffd4486f-7zxcv": networkPlugin cni failed to set up pod "win-webserver-fffd4486f-7zxcv_default" network: error while GETHNSNewtorkByName(vxlan0): Network vxlan0 not found
Hi,
i followed the installation procedure from https://v1-17.docs.kubernetes.io/docs/setup/production-environment/windows/user-guide-windows-nodes/
I'm able to create pods but they're not able to reach the service network (10.96.0.0/12)
I read that ping doesn't work by design but i'm not able to reach it through Test-NetConnection
I tried different version of docker, windows, kubernetes, flannel & cni plugin.
I'm using 3 VMs (Linux master, linux worker and windows worker) on an Hyper-V workstation.
Spoofing is enabled.
Linux version is CentOS 7
No issue with l2bridge but i can't use it on the target infrastructure.
any ideas?
Thank you
As a kubeadm maintainers team, we would like to have a signal wrt bootstrap windows nodes with kubeadm in the kubeadm test grid
There are logistic to be addressed, but this issue aims to start a joint effort for getting this in place hopefully during the 1.17 cycle
/sig cluster-lifecycle
/cc @neolit123
PS C:\k> kubectl get no
NAME STATUS ROLES AGE VERSION
m1 Ready master 19d v1.18.1
m2 Ready master 19d v1.18.1
n1 Ready <none> 119m v1.18.1
w1 Ready <none> 3h35m v1.18.1
PS C:\k> kubectl get po -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-66bff467f8-l4h2x 1/1 Running 0 114m
coredns-66bff467f8-n7994 1/1 Running 0 115m
etcd-m1 1/1 Running 3584 19d
etcd-m2 1/1 Running 155 19d
kube-apiserver-m1 1/1 Running 4285 19d
kube-apiserver-m2 1/1 Running 140 19d
kube-controller-manager-m1 1/1 Running 27 19d
kube-controller-manager-m2 1/1 Running 27 19d
kube-flannel-ds-amd64-h6wlc 1/1 Running 6 10d
kube-flannel-ds-amd64-lrf77 1/1 Running 5 10d
kube-flannel-ds-amd64-nh5bv 1/1 Running 0 120m
kube-flannel-ds-windows-amd64-hp8sj 1/1 Running 1 3h15m
kube-proxy-fs47v 1/1 Running 12 19d
kube-proxy-windows-s5tk5 0/1 ContainerCreating 0 44m
kube-proxy-z8dmn 1/1 Running 0 120m
kube-proxy-zgl5q 1/1 Running 11 19d
kube-scheduler-m1 1/1 Running 28 19d
kube-scheduler-m2 1/1 Running 31 19d
[root@m1 ~]# kubectl describe -n kube-system po kube-proxy-windows-s5tk5
Name: kube-proxy-windows-s5tk5
Namespace: kube-system
Priority: 0
Node: w1/10.7.1.221
Start Time: Sat, 02 May 2020 03:10:26 -0400
Labels: controller-revision-hash=559f567df8
k8s-app=kube-proxy-windows
pod-template-generation=1
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: DaemonSet/kube-proxy-windows
Containers:
kube-proxy:
Container ID:
Image: sigwindowstools/kube-proxy:v1.18.1
Image ID:
Port: <none>
Host Port: <none>
Command:
powershell
Args:
-file
/var/lib/kube-proxy-windows/run-script.ps1
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment:
NODE_NAME: (v1:spec.nodeName)
POD_IP: (v1:status.podIP)
Mounts:
/host from host (rw)
/var/lib/kube-proxy from kube-proxy (rw)
/var/lib/kube-proxy-windows from kube-proxy-windows (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-proxy-token-s46nq (ro)
\\.\pipe\rancher_wins from wins (rw)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
kube-proxy-windows:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kube-proxy-windows
Optional: false
kube-proxy:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: kube-proxy
Optional: false
host:
Type: HostPath (bare host directory volume)
Path: /
HostPathType:
wins:
Type: HostPath (bare host directory volume)
Path: \\.\pipe\rancher_wins
HostPathType:
kube-proxy-token-s46nq:
Type: Secret (a volume populated by a Secret)
SecretName: kube-proxy-token-s46nq
Optional: false
QoS Class: BestEffort
Node-Selectors: kubernetes.io/os=windows
Tolerations:
CriticalAddonsOnly
node.kubernetes.io/disk-pressure:NoSchedule
node.kubernetes.io/memory-pressure:NoSchedule
node.kubernetes.io/not-ready:NoExecute
node.kubernetes.io/pid-pressure:NoSchedule
node.kubernetes.io/unreachable:NoExecute
node.kubernetes.io/unschedulable:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned kube-system/kube-proxy-windows-s5tk5 to w1
Warning FailedCreatePodSandBox 49m kubelet, w1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "803b527affc367a843017a49b3b796290e4c303c762e70ed1b03559f44ebcf5a" network for pod "kube-proxy-windows-s5tk5": networkPlugin cni failed to set up pod "kube-proxy-windows-s5tk5_kube-system" network: open /run/flannel/subnet.env: The system cannot find the path specified.
Warning FailedCreatePodSandBox 49m kubelet, w1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "bd24c17bbf37ffc2710ee095daa31bee62bf70128a1f9387c0dec066820f8f5c" network for pod "kube-proxy-windows-s5tk5": networkPlugin cni failed to set up pod "kube-proxy-windows-s5tk5_kube-system" network: open /run/flannel/subnet.env: The system cannot find the path specified.
Warning FailedCreatePodSandBox 49m kubelet, w1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "c172774f4be8f67ed34db4418aafeb61c98c5bef18f5e01060c4d5a498951279" network for pod "kube-proxy-windows-s5tk5": networkPlugin cni failed to set up pod "kube-proxy-windows-s5tk5_kube-system" network: open /run/flannel/subnet.env: The system cannot find the path specified.
Warning FailedCreatePodSandBox 49m kubelet, w1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "765e0723190a180ddb904305c33aedb69705442ae21e5cae89120bfca5a1b561" network for pod "kube-proxy-windows-s5tk5": networkPlugin cni failed to set up pod "kube-proxy-windows-s5tk5_kube-system" network: open /run/flannel/subnet.env: The system cannot find the path specified.
Warning FailedCreatePodSandBox 49m kubelet, w1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "23558bbe4f32fbec6bc45f334711998b42d0cb21dc8710679fe128401cffa6f5" network for pod "kube-proxy-windows-s5tk5": networkPlugin cni failed to set up pod "kube-proxy-windows-s5tk5_kube-system" network: open /run/flannel/subnet.env: The system cannot find the path specified.
Warning FailedCreatePodSandBox 49m kubelet, w1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "4a2a22914d135a2efe83fc2d06ba165d74586f3aebde5a94bc746aee4d15b5ff" network for pod "kube-proxy-windows-s5tk5": networkPlugin cni failed to set up pod "kube-proxy-windows-s5tk5_kube-system" network: open /run/flannel/subnet.env: The system cannot find the path specified.
Warning FailedCreatePodSandBox 49m kubelet, w1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "b22ba71d9f06f01594d78cf527735cd7157f5e8037519a7d79e16666f07e1842" network for pod "kube-proxy-windows-s5tk5": networkPlugin cni failed to set up pod "kube-proxy-windows-s5tk5_kube-system" network: open /run/flannel/subnet.env: The system cannot find the path specified.
Warning FailedCreatePodSandBox 49m kubelet, w1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "260364f6a138106a725c5b6205db9d050a03e75a8f5075aff313c338041d3553" network for pod "kube-proxy-windows-s5tk5": networkPlugin cni failed to set up pod "kube-proxy-windows-s5tk5_kube-system" network: open /run/flannel/subnet.env: The system cannot find the path specified.
Warning FailedCreatePodSandBox 49m kubelet, w1 Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "080dbb1b0bd098baa9c6131f32ca0778aa1579b93939ee455cec0ee0a1a3d214" network for pod "kube-proxy-windows-s5tk5": networkPlugin cni failed to set up pod "kube-proxy-windows-s5tk5_kube-system" network: open /run/flannel/subnet.env: The system cannot find the path specified.
Warning FailedCreatePodSandBox 9m18s (x840 over 48m) kubelet, w1 (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "41646bf0e43d2e12c65f8dc769bc03f8449d911dad7112dd072ebbe91ef16a21" network for pod "kube-proxy-windows-s5tk5": networkPlugin cni failed to set up pod "kube-proxy-windows-s5tk5_kube-system" network: open /run/flannel/subnet.env: The system cannot find the path specified.
Normal SandboxChanged 4m18s (x956 over 49m) kubelet, w1 Pod sandbox changed, it will be killed and re-created.
I used the latest binaries of kubernetes and the PrepareNodes script.
But I still find logs as Failed to list *v1.Node: Get https://10.246.44.131:6443/api/v1/nodes?resourceVersion=0: http2: no cached connection was available in flannel pod. Below is the complet log:
[root@xxx-centos-7 hzxieqianyue]# kubectl logs kube-flannel-ds-windows-amd64-482m2 -n kube-system
Directory: C:\host\etc\cni
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 5/6/2020 4:25 PM net.d
Directory: C:\host\etc
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 5/9/2020 2:23 PM kube-flannel
Directory: C:\host\opt\cni
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 5/6/2020 4:25 PM bin
Directory: C:\host\k
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 5/9/2020 11:42 AM flannel
Directory: C:\host\k\flannel\var\run\secrets\kubernetes.io
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 5/9/2020 11:36 AM serviceaccount
WARNING: The names of some imported commands from the module 'hns' include unapproved verbs that might make them less
discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose
parameter. For a list of approved verbs, type Get-Verb.
Invoke-HnsRequest : @{Error=提供的策略配置无效或缺少参数。 ; ErrorCode=2151350285; Success=False}
At C:\k\flannel\hns.psm1:233 char:16
+ ... return Invoke-HnsRequest -Method POST -Type networks -Data $Json ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Invoke-HNSRequest
I0509 14:24:07.064568 7864 main.go:518] Determining IP address of default interface
I0509 14:24:09.032716 7864 main.go:531] Using interface with name Ethernet and address 10.246.44.130
I0509 14:24:09.032716 7864 main.go:548] Defaulting external address to interface address (10.246.44.130)
I0509 14:24:09.065974 7864 kube.go:119] Waiting 10m0s for node controller to sync
I0509 14:24:09.065974 7864 kube.go:306] Starting kube subnet manager
I0509 14:24:10.066040 7864 kube.go:126] Node controller sync successful
I0509 14:24:10.066040 7864 main.go:246] Created subnet manager: Kubernetes Subnet Manager - win-3fjh9ve50cq
I0509 14:24:10.066040 7864 main.go:249] Installing signal handlers
I0509 14:24:10.066956 7864 main.go:390] Found network config - Backend type: host-gw
I0509 14:24:10.066956 7864 hostgw_windows.go:73] HOST-GW config: {Name:cbr0 DNSServerList:}
I0509 14:24:10.102921 7864 hostgw_windows.go:157] Attempting to create HNSNetwork {"Name":"cbr0","Type":"L2Bridge","Subnets":[{"AddressPrefix":"10.244.5.0/24","GatewayAddress":"10.244.5.1"}]}
E0509 14:24:11.044412 7864 streamwatcher.go:109] Unable to decode an event from the watch stream: read tcp 10.246.44.130:51336->10.246.44.131:6443: wsarecv: An established connection was aborted by the software in your host machine.
E0509 14:24:11.044412 7864 reflector.go:304] github.com/coreos/flannel/subnet/kube/kube.go:307: Failed to watch *v1.Node: Get https://10.246.44.131:6443/api/v1/nodes?resourceVersion=2342170&timeoutSeconds=582&watch=true: http2: no cached connection was available
E0509 14:24:12.053470 7864 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:307: Failed to list *v1.Node: Get https://10.246.44.131:6443/api/v1/nodes?resourceVersion=0: http2: no cached connection was available
I0509 14:24:12.221469 7864 hostgw_windows.go:164] Waiting to get ManagementIP from HNSNetwork cbr0
I0509 14:24:12.736052 7864 hostgw_windows.go:174] Waiting to get net interface for HNSNetwork cbr0 (10.246.44.130)
E0509 14:24:13.064714 7864 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:307: Failed to list *v1.Node: Get https://10.246.44.131:6443/api/v1/nodes?resourceVersion=0: http2: no cached connection was available
E0509 14:24:14.073672 7864 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:307: Failed to list *v1.Node: Get https://10.246.44.131:6443/api/v1/nodes?resourceVersion=0: http2: no cached connection was available
E0509 14:24:15.073734 7864 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:307: Failed to list *v1.Node: Get https://10.246.44.131:6443/api/v1/nodes?resourceVersion=0: http2: no cached connection was available
I0509 14:24:15.444317 7864 hostgw_windows.go:183] Created HNSNetwork cbr0
I0509 14:24:15.448222 7864 hostgw_windows.go:212] Attempting to create bridge HNSEndpoint &{Id: Name:cbr0_ep VirtualNetwork:8D90C0B3-1A58-4C7D-BDE5-73DE0B955173 VirtualNetworkName: Policies:[] MacAddress: IPAddress:10.244.5.2 DNSSuffix: DNSServerList: GatewayAddress: EnableInternalDNS:false DisableICC:false PrefixLength:0 IsRemoteEndpoint:false EnableLowMetric:false Namespace:<nil> EncapOverhead:0}
I0509 14:24:15.466900 7864 hostgw_windows.go:217] Created bridge HNSEndpoint cbr0_ep
I0509 14:24:15.466900 7864 hostgw_windows.go:221] Waiting to attach bridge endpoint cbr0_ep to host
E0509 14:24:16.074559 7864 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:307: Failed to list *v1.Node: Get https://10.246.44.131:6443/api/v1/nodes?resourceVersion=0: http2: no cached connection was available
E0509 14:24:17.074680 7864 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:307: Failed to list *v1.Node: Get https://10.246.44.131:6443/api/v1/nodes?resourceVersion=0: http2: no cached connection was available
E0509 14:24:18.075379 7864 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:307: Failed to list *v1.Node: Get https://10.246.44.131:6443/api/v1/nodes?resourceVersion=0: http2: no cached connection was available
I0509 14:24:18.542661 7864 hostgw_windows.go:229] Attached bridge endpoint cbr0_ep to host successfully
I0509 14:24:18.907815 7864 hostgw_windows.go:237] Found {Idx:20 Name:vEthernet (Ethernet) 2 InterfaceMetric:25 DhcpEnabled:false IpAddress:10.246.44.130 SubnetPrefix:24 GatewayMetric:256 DefaultGatewayAddress:10.246.44.1} interface with IP 10.246.44.130
I0509 14:24:19.042572 7864 hostgw_windows.go:249] Enabled forwarding on vEthernet (Ethernet) 2 index 20
E0509 14:24:19.075967 7864 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:307: Failed to list *v1.Node: Get https://10.246.44.131:6443/api/v1/nodes?resourceVersion=0: http2: no cached connection was available
I0509 14:24:19.244242 7864 hostgw_windows.go:237] Found {Idx:45 Name:vEthernet (cbr0_ep) InterfaceMetric:25 DhcpEnabled:false IpAddress:10.244.5.2 SubnetPrefix:24 GatewayMetric:256 DefaultGatewayAddress:10.244.5.1} interface with IP 10.244.5.2
I0509 14:24:19.355554 7864 hostgw_windows.go:249] Enabled forwarding on vEthernet (cbr0_ep) index 45
I0509 14:24:19.355554 7864 main.go:313] Changing default FORWARD chain policy to ACCEPT
I0509 14:24:19.363308 7864 main.go:321] Wrote subnet file to /run/flannel/subnet.env
I0509 14:24:19.363308 7864 main.go:325] Running backend.
I0509 14:24:19.363308 7864 main.go:343] Waiting for all goroutines to exit
I0509 14:24:19.363308 7864 route_network_windows.go:51] Watching for new subnet leases
I0509 14:24:19.372155 7864 route_network_windows.go:94] Subnet added: 10.244.0.0/24 via 10.246.44.131
E0509 14:24:20.076134 7864 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:307: Failed to list *v1.Node: Get https://10.246.44.131:6443/api/v1/nodes?resourceVersion=0: http2: no cached connection was available
E0509 14:24:21.076326 7864 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:307: Failed to list *v1.Node: Get https://10.246.44.131:6443/api/v1/nodes?resourceVersion=0: http2: no cached connection was available
E0509 14:24:22.077035 7864 reflector.go:201] github.com/coreos/flannel/subnet/kube/kube.go:307: Failed to list *v1.Node: Get https://10.246.44.131:6443/api/v1/nodes?resourceVersion=0: http2: no cached connection was available
I posted a comment here under the issue #37.
And submitted a review in #38. I have a question for the code below:
func setupL2bridge(interfaceName string) {
run(fmt.Sprintf(`ipmo C:\k\flannel\hns.psm1; New-HNSNetwork -Type Overlay -AddressPrefix "192.168.255.0/30"`+
-Type Overlay here shouldn't be -Type l2bridge?
EDIT: we had discussions on slack:
https://kubernetes.slack.com/archives/C0SJ4AFB7/p1584740410152900
and the network failures are apparently caused by a Windows bug when executing 32bit apps on 64bit images that have PATCH level version differences with the host, that should be resolved once we have the latest agnhost image at the e2eteam account:
https://hub.docker.com/u/e2eteam
latest kubeadm e2e test run reports the following failures.
some of them might be flakes.
https://k8s-testgrid.appspot.com/sig-windows#kubeadm-windows-gcp-k8s-stable
[k8s.io] Container Lifecycle Hook when create a pod with lifecycle hook should execute poststart http hook properly [NodeConformance] [Conformance]
TODO
[k8s.io] Container Lifecycle Hook when create a pod with lifecycle hook should execute prestop http hook properly [NodeConformance] [Conformance]
TODO
[k8s.io] Docker Containers should use the image defaults if command and args are blank [NodeConformance] [Conformance]
the test is here:
https://github.com/kubernetes/kubernetes/blob/master/test/e2e/common/docker_containers.go#L33-L50
this could be caused by flake when polling to fetch the logs, or simply because the Agnhost container for windows is not printing "Paused"?
[sig-api-machinery] Aggregator Should be able to support the 1.10 Sample API Server using the current Aggregator [Conformance]
this one is now disabled via a SKIP=...Aggregator |....
#55
[sig-network] Services should be able to change the type from ExternalName to ClusterIP [Conformance]
TODO
[sig-network] Services should be able to change the type from ExternalName to NodePort [Conformance]
TODO
[sig-network] Services should be able to create a functioning NodePort service [Conformance]
TODO
[sig-network][sig-windows] Networking Granular Checks: Pods should function for node-pod communication: udp
for the sig-network* failures we are suspecting a potential agnhost issue, because benmoss verified that the underlying test details work with a simple web server.
[sig-storage] HostPath should support r/w [NodeConformance]
this might be caused by a missing /tmp on the windows nodes:
https://github.com/kubernetes/kubernetes/blob/1bb68a2cde08e39875f8e4f53bf429e14438835e/test/e2e/common/host_path.go#L64-L89
/kind failing-test
/priority important-soon
/assign @neolit123 @benmoss
Question:
Does Docker for Windows Server 2019 support GUI-based applications?
I want to run a GUI application desktop(With a 3d map inside) in windows container at windows server host.
Hi,
I recently tried to deployed an hybrid cluster, I have a setup with Kubeadm 1.18.5, I have been using this guide:
https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/adding-windows-nodes/
I'm running windows server 2019 with the following feature enabled:
I have deployed flannel and flannel for windows as long as kube-proxy for windows.
Network connectivity seems fine between linux and windows pods, also dns resolution with kube-proxy cluster IP is also OK.
My only problem is that I cannot access the internet I have one error when starting flannel:
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 7/13/2020 12:49 PM serviceaccount
WARNING: The names of some imported commands from the module 'hns' include unapproved verbs that might make them less
discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose
parameter. For a list of approved verbs, type Get-Verb.
Invoke-HnsRequest : @{Error=An adapter was not found. ; ErrorCode=2151350278; Success=False}
At C:\k\flannel\hns.psm1:233 char:16
+ ... return Invoke-HnsRequest -Method POST -Type networks -Data $Json ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Invoke-HNSRequest
FATA[2020-07-13T12:53:32Z] rpc error: code = Internal desc = could not create IP forward entry: The object already exists.
I have tried changin the interface name form "Ethernet" to "Ethernet 3", I also tried to rename the adapter to "wired" but no use I still get the same error.
Also, I'm on AWS, with a single AZ setup and src/dest check disabled.
Here are the full flannel logs
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 7/13/2020 12:05 PM net.d
Directory: C:\host\etc
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 7/13/2020 12:05 PM kube-flannel
Directory: C:\host\opt\cni
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 7/13/2020 12:05 PM bin
Directory: C:\host\k
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 7/13/2020 12:49 PM flannel
Directory: C:\host\k\flannel\var\run\secrets\kubernetes.io
Mode LastWriteTime Length Name
---- ------------- ------ ----
d----- 7/13/2020 12:49 PM serviceaccount
WARNING: The names of some imported commands from the module 'hns' include unapproved verbs that might make them less
discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose
parameter. For a list of approved verbs, type Get-Verb.
Invoke-HnsRequest : @{Error=An adapter was not found. ; ErrorCode=2151350278; Success=False}
At C:\k\flannel\hns.psm1:233 char:16
+ ... return Invoke-HnsRequest -Method POST -Type networks -Data $Json ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Write-Error], WriteErrorException
+ FullyQualifiedErrorId : Microsoft.PowerShell.Commands.WriteErrorException,Invoke-HNSRequest
FATA[2020-07-13T12:53:32Z] rpc error: code = Internal desc = could not create IP forward entry: The object already exists.
I0713 12:53:32.972594 6772 main.go:518] Determining IP address of default interface
I0713 12:53:33.264677 6772 main.go:531] Using interface with name vEthernet (Ethernet 3) and address 172.31.5.66
I0713 12:53:33.264677 6772 main.go:548] Defaulting external address to interface address (172.31.5.66)
I0713 12:53:33.311346 6772 kube.go:119] Waiting 10m0s for node controller to sync
I0713 12:53:33.311346 6772 kube.go:306] Starting kube subnet manager
I0713 12:53:34.313338 6772 kube.go:126] Node controller sync successful
I0713 12:53:34.313338 6772 main.go:246] Created subnet manager: Kubernetes Subnet Manager - ec2amaz-susd876
I0713 12:53:34.313338 6772 main.go:249] Installing signal handlers
I0713 12:53:34.313338 6772 main.go:390] Found network config - Backend type: host-gw
I0713 12:53:34.313338 6772 hostgw_windows.go:73] HOST-GW config: {Name:cbr0 DNSServerList:}
I0713 12:53:34.333371 6772 hostgw_windows.go:125] Found existing HNSNetwork cbr0
I0713 12:53:34.338360 6772 hostgw_windows.go:192] Found existing bridge HNSEndpoint cbr0_ep
I0713 12:53:34.338360 6772 hostgw_windows.go:221] Waiting to attach bridge endpoint cbr0_ep to host
I0713 12:53:34.857310 6772 hostgw_windows.go:229] Attached bridge endpoint cbr0_ep to host successfully
I0713 12:53:35.062855 6772 hostgw_windows.go:237] Found {Idx:19 Name:vEthernet (Ethernet 3) InterfaceMetric:15 DhcpEnabled:true IpAddress:172.31.5.66 SubnetPrefix:20 GatewayMetric:0 DefaultGatewayAddress:172.31.0.1} interface with IP 172.31.5.66
I0713 12:53:35.150411 6772 hostgw_windows.go:249] Enabled forwarding on vEthernet (Ethernet 3) index 19
I0713 12:53:35.320552 6772 hostgw_windows.go:237] Found {Idx:41 Name:vEthernet (cbr0_ep) InterfaceMetric:15 DhcpEnabled:false IpAddress:10.244.8.2 SubnetPrefix:24 GatewayMetric:256 DefaultGatewayAddress:10.244.8.1} interface with IP 10.244.8.2
I0713 12:53:35.393418 6772 hostgw_windows.go:249] Enabled forwarding on vEthernet (cbr0_ep) index 41
I0713 12:53:35.393418 6772 main.go:313] Changing default FORWARD chain policy to ACCEPT
I0713 12:53:35.405680 6772 main.go:321] Wrote subnet file to /run/flannel/subnet.env
I0713 12:53:35.405680 6772 main.go:325] Running backend.
I0713 12:53:35.405680 6772 main.go:343] Waiting for all goroutines to exit
I0713 12:53:35.405680 6772 route_network_windows.go:51] Watching for new subnet leases
I0713 12:53:35.415560 6772 route_network_windows.go:94] Subnet added: 10.244.4.0/24 via 172.31.12.160
I0713 12:53:36.779553 6772 route_network_windows.go:94] Subnet added: 10.244.7.0/24 via 172.31.3.190
I0713 12:53:37.905706 6772 route_network_windows.go:94] Subnet added: 10.244.2.0/24 via 172.31.11.135
I0713 12:53:39.046471 6772 route_network_windows.go:94] Subnet added: 10.244.1.0/24 via 172.31.4.180
I0713 12:53:40.185520 6772 route_network_windows.go:94] Subnet added: 10.244.0.0/24 via 172.31.7.42
I0713 12:53:41.349307 6772 route_network_windows.go:94] Subnet added: 10.244.3.0/24 via 172.31.6.96
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.