kubeop/k8s

Deploy a Production Ready Kubernetes High Availability Cluster with Binary

支持的发型版

• CentOS/RHEL 7，8，9
• AlmaLinux 8，9
• RockyLinux 8，9
• Ubuntu Server 20.04，22.04
• Debian 12

支持组件

• Core
  • kubernetes
  • etcd
  • containerd
• Network Plugin
  • cni-plugins
  • calico
  • cilium
  • flanneld
  • kube-router
  • macvlan
• Application
  • coredns
  • node-local-dns
  • metrics-server
  • nvidia_device_plugin

前期配置

安装Ansible

请根据下表安装合适的Python版本和Ansible版本

Python	Ansible
>=3.9	>=2.15.5

安装Ansible示例

pip3 install ansible -i https://mirrors.ustc.edu.cn/pypi/web/simple
pip3 install netaddr -i https://mirrors.ustc.edu.cn/pypi/web/simple

• 请使用对应Python版本的pip安装ansible和netaddr。
• 控制节点和被控节点Python版本尽量保持一致，否则执行可能出现问题。
• 更详细的版本支持矩阵，请参考：https://docs.ansible.com/ansible-core/devel/reference_appendices/release_and_maintenance.html#ansible-core-support-matrix

修改 inventory

请按照inventory模板格式修改对应资源

• 当haproxy和kube-apiserver部署在同一台服务器时，请确保端口不冲突。

配置 group_vars

编辑group_vars/all.yml文件，根据自己的实际环境进行配置。

请注意：

• Kubernetes 的最低版本要求为 v1.26

• 请尽量将etcd安装在独立的服务器上，不建议跟master安装在一起。数据盘尽量使用SSD盘。

• Pod 和Service IP网段建议使用保留私有IP段，建议（Pod IP不与Service IP重复，也不要与主机IP段重复，同时也避免与docker0网卡的网段冲突。）：

  • Pod 网段
    • A类地址：10.0.0.0/8
    • B类地址：172.16-31.0.0/12-16
    • C类地址：192.168.0.0/16
  • Service网段
    • A类地址：10.0.0.0/16-24
    • B类地址：172.16-31.0.0/16-24
    • C类地址：192.168.0.0/16-24

• 如是离线环境，提前将相关包下载放到内网下载服务器，然后将groups/all.yml替换为内网下载地址即可（确保可以使用yum/apt/dnf等安装系统依赖包）

挂载数据盘

如已经自行格式化并挂载目录，可以跳过此步骤。

ansible-playbook fdisk.yml -i inventory -e "disk=sdb dir=/data"

如果是NVME的磁盘，请使用以下方式:

ansible-playbook fdisk.yml -i inventory -e "disk=sdb dir=/data type=nvme"

⚠️：

• 此脚本会格式化{{disk}}指定的硬盘，并挂载到{{dir}}目录。
• 会将/var/lib/etcd、/var/lib/containerd、/var/lib/kubelet、/var/log/pods数据目录绑定到此数据盘{{dir}}/containers/etcd、{{dir}}/containers/containerd、{{dir}}/containers/kubelet、{{dir}}/containers/pods目录，以达到多个数据目录共用一个数据盘，而无需修改kubernetes相关数据目录。

如需不同目录挂载不同数据盘，可以使用以下命令单独挂载

ansible-playbook fdisk.yml -i inventory -l master -e "disk=sdb dir=/var/lib/etcd" --skip-tags=bind_dir

如已经格式化并挂载过数据盘，可以使用以下命令将数据目录绑定到数据盘

ansible-playbook fdisk.yml -i inventory -l master -e "disk=sdb dir=/data" -t bind_dir

下载离线包

# 如从自建文件服务器，请修改roles/download/defaults/main.yml文件中的默认地址
ansible-playbook cluster.yml -i inventory -t download

• 请确保Ansible控制端可以访问Internet，否则无法下载离线安装包。
• 在其他Internet节点下载后，按照要求目录结构拷贝到{{ download.dest }}目录中也可。

同步镜像

# 如集群节点可以连接公网，可以跳过此步骤。
# 如不能连接公网或需使用私有镜像仓库，请自行同步group_vars/all.yml中定义的镜像至私有镜像仓库。
# 也可以使用 https://github.com/AliyunContainerService/image-syncer/releases 同步

部署集群

# 如未执行下载，可以执行以下命令
ansible-playbook cluster.yml -i inventory

# 如已执行下载离线包，可以跳过下载
ansible-playbook cluster.yml -i inventory --skip-tags=download

如是公有云环境，使用公有云的负载均衡即可（需提前配置好负载均衡），无需安装haproxy和keepalived。

ansible-playbook cluster.yml -i inventory --skip-tags=haproxy,keepalived

• 默认会对节点进行初始化操作，集群节点会取主机名最后两段和IP作为集群节点名称。

如果想让master节点也进行调度，可以添加使用以下方式

ansible-playbook cluster.yml -i inventory --skip-tags=create_master_taint

扩容节点

扩容master节点

扩容时，建议注释inventory文件master组中旧服务器信息，仅保留扩容节点的信息。

格式化挂载数据盘

ansible-playbook fdisk.yml -i inventory -l ${SCALE_MASTER_IP} -e "disk=sdb dir=/data"

执行生成节点证书

ansible-playbook cluster.yml -i inventory -t cert

执行节点初始化

ansible-playbook cluster.yml -i inventory -l ${SCALE_MASTER_IP} -t verify,init

执行节点扩容

ansible-playbook cluster.yml -i inventory -l ${SCALE_MASTER_IP} -t master,containerd,worker --skip-tags=bootstrap,create_worker_label

扩容worker节点

扩容时，建议注释inventory文件worker组中旧服务器信息，仅保留扩容节点的信息。

格式化挂载数据盘

ansible-playbook fdisk.yml -i inventory -l ${SCALE_MASTER_IP} -e "disk=sdb dir=/data"

执行生成节点证书

ansible-playbook cluster.yml -i inventory -t cert

执行节点初始化

ansible-playbook cluster.yml -i inventory -l ${SCALE_WORKER_IP} -t verify,init

执行节点扩容

ansible-playbook cluster.yml -i inventory -l ${SCALE_WORKER_IP} -t containerd,worker --skip-tags=bootstrap,create_master_label

替换集群证书

先备份并删除证书目录{{cert.dir}}，重新创建{{cert.dir}}，并将token、sa.pub、sa.key文件拷贝至新创建的{{cert.dir}}（这三个文件务必保留，不能更改），然后执行以下步骤重新生成证书并分发证书。

ansible-playbook cluster.yml -i inventory -t cert,dis_certs

然后依次重启每个节点。

重启etcd

ansible -i inventory etcd -m systemd -a "name=etcd state=restarted"

验证etcd

etcdctl endpoint health \
        --cacert=/etc/etcd/pki/etcd-ca.pem \
        --cert=/etc/etcd/pki/etcd-healthcheck-client.pem \
        --key=/etc/etcd/pki/etcd-healthcheck-client.key \
        --endpoints=https://172.16.90.101:2379,https://172.16.90.102:2379,https://172.16.90.103:2379

逐个删除旧的kubelet证书

ansible -i inventory master,worker -m shell -a "rm -rf /etc/kubernetes/pki/kubelet*"

• -l参数更换为具体节点IP。

逐个重启节点

ansible-playbook cluster.yml -i inventory -l ${IP} -t restart_apiserver,restart_controller,restart_scheduler,restart_kubelet,restart_proxy,healthcheck

• 如calico、metrics-server等服务也使用了集群证书，请记得一起更新相关证书。
• -l参数更换为具体节点IP。

重启网络插件

kubectl get pod -n kube-system | grep -v NAME | grep cilium | awk '{print $1}' | xargs kubectl -n kube-system delete pod

• 更新证书可能会导致网络插件异常，建议重启。
• 示例为重启cilium插件命令，请根据不同网络插件自行替换。

升级kubernetes版本

请先编辑group_vars/all.yml，修改kubernetes.version为新版本。

下载新版本安装包

ansible-playbook cluster.yml -i inventory -t download

安装kubernetes组件

ansible-playbook cluster.yml -i inventory -t install_kubectl,install_master,install_worker

更新配置文件

ansible-playbook cluster.yml -i inventory -t dis_master_config,dis_worker_config

然后依次重启每个kubernetes组件。

ansible-playbook cluster.yml -i inventory -l ${IP} -t restart_apiserver,restart_controller,restart_scheduler,restart_kubelet,restart_proxy,healthcheck

• -l参数更换为具体节点IP。

清理worker节点

ansible-playbook reset.yml -i inventory -l ${IP} -e "flush_iptables=true enable_dual_stack_networks=false"

k8s's Issues

这两行重复了?

https://github.com/k8sre/k8s/blob/a6a81f8f8e55f1fbab4a692c5f13f48f995e51c1/roles/master/templates/kube-controller-manager.conf.j2#L5

Failed to Create taint for control-plane

TASK [init : Install base application] *************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.101]
ok: [192.168.137.11]
ok: [192.168.137.12]
ok: [192.168.137.10]

TASK [init : Install base application] *************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.11]
ok: [192.168.137.12]
ok: [192.168.137.10]
ok: [192.168.137.101]

TASK [init : Install base application] *************************************************************************************************************************************************************************************************************************************************************************************
skipping: [192.168.137.11]
skipping: [192.168.137.12]
skipping: [192.168.137.10]
skipping: [192.168.137.101]

PLAY [Set Haproxy + Keepalived] ********************************************************************************************************************************************************************************************************************************************************************************************

PLAY [Set Etcd Cluster] ****************************************************************************************************************************************************************************************************************************************************************************************************

TASK [etcd : Create etcd user group] ***************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.11]
ok: [192.168.137.10]
ok: [192.168.137.12]

TASK [etcd : Create etcd user] *********************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.12]
ok: [192.168.137.11]

TASK [etcd : Download etcd] ************************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]

TASK [etcd : Install etcd] *************************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.11] => (item={'src': '/tmp/etcd-v3.5.7-linux-amd64/etcd'})
ok: [192.168.137.10] => (item={'src': '/tmp/etcd-v3.5.7-linux-amd64/etcd'})
ok: [192.168.137.12] => (item={'src': '/tmp/etcd-v3.5.7-linux-amd64/etcd'})
ok: [192.168.137.10] => (item={'src': '/tmp/etcd-v3.5.7-linux-amd64/etcdctl'})
ok: [192.168.137.11] => (item={'src': '/tmp/etcd-v3.5.7-linux-amd64/etcdctl'})
ok: [192.168.137.12] => (item={'src': '/tmp/etcd-v3.5.7-linux-amd64/etcdctl'})
ok: [192.168.137.10] => (item={'src': '/tmp/etcd-v3.5.7-linux-amd64/etcdutl'})
ok: [192.168.137.12] => (item={'src': '/tmp/etcd-v3.5.7-linux-amd64/etcdutl'})
ok: [192.168.137.11] => (item={'src': '/tmp/etcd-v3.5.7-linux-amd64/etcdutl'})

TASK [etcd : Distribution certs] *******************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10] => (item={'line': '/opt/certs/etcd-ca.pem'})
ok: [192.168.137.11] => (item={'line': '/opt/certs/etcd-ca.pem'})
ok: [192.168.137.12] => (item={'line': '/opt/certs/etcd-ca.pem'})
ok: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/etcd-server.pem'})
ok: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/etcd-server.pem'})
ok: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/etcd-server.pem'})
ok: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/etcd-server.key'})
ok: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/etcd-server.key'})
ok: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/etcd-server.key'})
ok: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/etcd-peer.pem'})
ok: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/etcd-peer.pem'})
ok: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/etcd-peer.pem'})
ok: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/etcd-peer.key'})
ok: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/etcd-peer.key'})
ok: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/etcd-peer.key'})
ok: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/etcd-healthcheck-client.pem'})
ok: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/etcd-healthcheck-client.pem'})
ok: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/etcd-healthcheck-client.pem'})
ok: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/etcd-healthcheck-client.key'})
ok: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/etcd-healthcheck-client.key'})
ok: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/etcd-healthcheck-client.key'})

TASK [etcd : Create etcd data directory] ***********************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]

TASK [etcd : Generate etcd configure] **************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]

PLAY [Set Control Plane Nodes] *********************************************************************************************************************************************************************************************************************************************************************************************

TASK [master : Create kubernetes user group] *******************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]

TASK [master : Create kubernetes user] *************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]

TASK [master : Create kubernetes config directory] *************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10] => (item=~/.kube)
ok: [192.168.137.11] => (item=~/.kube)
ok: [192.168.137.12] => (item=~/.kube)
ok: [192.168.137.10] => (item=/etc/kubernetes/pki)
ok: [192.168.137.11] => (item=/etc/kubernetes/pki)
ok: [192.168.137.12] => (item=/etc/kubernetes/pki)

TASK [master : Create kubernetes log directory] ****************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10] => (item=/var/log/kubernetes)
ok: [192.168.137.11] => (item=/var/log/kubernetes)
ok: [192.168.137.12] => (item=/var/log/kubernetes)
ok: [192.168.137.10] => (item=/usr/libexec/kubernetes)
ok: [192.168.137.11] => (item=/usr/libexec/kubernetes)
ok: [192.168.137.12] => (item=/usr/libexec/kubernetes)

TASK [master : Install kubectl] ********************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]

TASK [master : Install master] *********************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.11] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-apiserver'})
changed: [192.168.137.10] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-apiserver'})
changed: [192.168.137.12] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-apiserver'})
changed: [192.168.137.11] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-controller-manager'})
changed: [192.168.137.12] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-controller-manager'})
changed: [192.168.137.10] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-controller-manager'})
changed: [192.168.137.11] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-scheduler'})
changed: [192.168.137.10] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-scheduler'})
changed: [192.168.137.12] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-scheduler'})

TASK [master : Distribution master certs] **********************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10] => (item={'line': '/opt/certs/ca.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/ca.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/ca.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/ca.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/ca.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/ca.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/sa.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/sa.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/sa.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/sa.pub'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/sa.pub'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/sa.pub'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/etcd-ca.pem'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/etcd-ca.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/etcd-ca.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/apiserver.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/apiserver.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/apiserver.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/apiserver.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/apiserver.pem'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/apiserver.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/apiserver-etcd-client.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/apiserver-etcd-client.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/apiserver-etcd-client.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/apiserver-etcd-client.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/apiserver-etcd-client.pem'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/apiserver-etcd-client.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/apiserver-kubelet-client.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/apiserver-kubelet-client.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/apiserver-kubelet-client.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/apiserver-kubelet-client.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/apiserver-kubelet-client.pem'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/apiserver-kubelet-client.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/front-proxy-ca.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/front-proxy-ca.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/front-proxy-ca.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/front-proxy-ca.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/front-proxy-ca.pem'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/front-proxy-ca.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/front-proxy-client.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/front-proxy-client.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/front-proxy-client.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/front-proxy-client.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/front-proxy-client.pem'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/front-proxy-client.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/controller-manager.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/controller-manager.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/controller-manager.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/controller-manager.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/controller-manager.pem'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/controller-manager.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/scheduler.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/scheduler.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/scheduler.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/scheduler.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/scheduler.pem'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/scheduler.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/admin.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/admin.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/admin.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/admin.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/admin.pem'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/admin.pem'})

TASK [master : Get token-id] ***********************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.12]
changed: [192.168.137.11]

TASK [master : Distribution kubectl kubeconfig] ****************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.11]
changed: [192.168.137.12]

TASK [master : Distribution master kubeconfig] *****************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10] => (item={'src': 'controller-manager.kubeconfig.j2', 'dest': '/etc/kubernetes/controller-manager.kubeconfig'})
changed: [192.168.137.11] => (item={'src': 'controller-manager.kubeconfig.j2', 'dest': '/etc/kubernetes/controller-manager.kubeconfig'})
changed: [192.168.137.12] => (item={'src': 'controller-manager.kubeconfig.j2', 'dest': '/etc/kubernetes/controller-manager.kubeconfig'})
changed: [192.168.137.10] => (item={'src': 'scheduler.kubeconfig.j2', 'dest': '/etc/kubernetes/scheduler.kubeconfig'})
changed: [192.168.137.11] => (item={'src': 'scheduler.kubeconfig.j2', 'dest': '/etc/kubernetes/scheduler.kubeconfig'})
changed: [192.168.137.12] => (item={'src': 'scheduler.kubeconfig.j2', 'dest': '/etc/kubernetes/scheduler.kubeconfig'})

TASK [master : Distribution master config] *********************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10] => (item={'src': 'kube-apiserver.conf.j2', 'dest': '/etc/kubernetes/kube-apiserver.conf'})
changed: [192.168.137.12] => (item={'src': 'kube-apiserver.conf.j2', 'dest': '/etc/kubernetes/kube-apiserver.conf'})
changed: [192.168.137.11] => (item={'src': 'kube-apiserver.conf.j2', 'dest': '/etc/kubernetes/kube-apiserver.conf'})
changed: [192.168.137.10] => (item={'src': 'kube-controller-manager.conf.j2', 'dest': '/etc/kubernetes/kube-controller-manager.conf'})
changed: [192.168.137.12] => (item={'src': 'kube-controller-manager.conf.j2', 'dest': '/etc/kubernetes/kube-controller-manager.conf'})
changed: [192.168.137.11] => (item={'src': 'kube-controller-manager.conf.j2', 'dest': '/etc/kubernetes/kube-controller-manager.conf'})
changed: [192.168.137.10] => (item={'src': 'kube-scheduler.conf.j2', 'dest': '/etc/kubernetes/kube-scheduler.conf'})
changed: [192.168.137.11] => (item={'src': 'kube-scheduler.conf.j2', 'dest': '/etc/kubernetes/kube-scheduler.conf'})
changed: [192.168.137.12] => (item={'src': 'kube-scheduler.conf.j2', 'dest': '/etc/kubernetes/kube-scheduler.conf'})
changed: [192.168.137.10] => (item={'src': 'audit-policy.yaml.j2', 'dest': '/etc/kubernetes/audit-policy.yaml'})
changed: [192.168.137.11] => (item={'src': 'audit-policy.yaml.j2', 'dest': '/etc/kubernetes/audit-policy.yaml'})
changed: [192.168.137.12] => (item={'src': 'audit-policy.yaml.j2', 'dest': '/etc/kubernetes/audit-policy.yaml'})

TASK [master : Distribution master systemd unit] ***************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10] => (item={'src': 'kube-apiserver.service.j2', 'dest': '/usr/lib/systemd/system/kube-apiserver.service'})
changed: [192.168.137.12] => (item={'src': 'kube-apiserver.service.j2', 'dest': '/usr/lib/systemd/system/kube-apiserver.service'})
changed: [192.168.137.11] => (item={'src': 'kube-apiserver.service.j2', 'dest': '/usr/lib/systemd/system/kube-apiserver.service'})
changed: [192.168.137.10] => (item={'src': 'kube-controller-manager.service.j2', 'dest': '/usr/lib/systemd/system/kube-controller-manager.service'})
changed: [192.168.137.12] => (item={'src': 'kube-controller-manager.service.j2', 'dest': '/usr/lib/systemd/system/kube-controller-manager.service'})
changed: [192.168.137.11] => (item={'src': 'kube-controller-manager.service.j2', 'dest': '/usr/lib/systemd/system/kube-controller-manager.service'})
changed: [192.168.137.10] => (item={'src': 'kube-scheduler.service.j2', 'dest': '/usr/lib/systemd/system/kube-scheduler.service'})
changed: [192.168.137.12] => (item={'src': 'kube-scheduler.service.j2', 'dest': '/usr/lib/systemd/system/kube-scheduler.service'})
changed: [192.168.137.11] => (item={'src': 'kube-scheduler.service.j2', 'dest': '/usr/lib/systemd/system/kube-scheduler.service'})

TASK [master : Restart kube-apiserver] *************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.11]
changed: [192.168.137.10]
changed: [192.168.137.12]

TASK [master : Restart kube-controller-manager] ****************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.11]
changed: [192.168.137.10]
changed: [192.168.137.12]

TASK [master : Restart kube-scheduler] *************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.12]
changed: [192.168.137.10]
changed: [192.168.137.11]

TASK [master : Add kubectl completion] *************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.12]
changed: [192.168.137.11]

TASK [master : Waiting kube-apiserver starting] ****************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.12]
ok: [192.168.137.11]

TASK [master : Waiting kube-controller-manager starting] *******************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]

TASK [master : Waiting kube-scheduler starting] ****************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]

TASK [master : Kube-apiserver health check] ********************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.12]
ok: [192.168.137.11]
ok: [192.168.137.10]

TASK [master : Kube-controller-manager health check] ***********************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]

TASK [master : Kube-scheduler health check] ********************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.12]
ok: [192.168.137.11]

PLAY [Set Container Runtime] ***********************************************************************************************************************************************************************************************************************************************************************************************

TASK [containerd : Install dependency] *************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.101]
changed: [192.168.137.10]
changed: [192.168.137.12]
changed: [192.168.137.11]

TASK [containerd : Install dependency] *************************************************************************************************************************************************************************************************************************************************************************************
skipping: [192.168.137.10]
skipping: [192.168.137.11]
skipping: [192.168.137.12]
skipping: [192.168.137.101]

TASK [containerd : Add nvidia-container-toolkit repository] ****************************************************************************************************************************************************************************************************************************************************************
skipping: [192.168.137.10] => (item={'name': 'libnvidia-container', 'description': 'libnvidia-container', 'baseurl': 'https://nvidia.github.io/libnvidia-container/stable/centos7/$basearch', 'gpgkey': 'https://nvidia.github.io/libnvidia-container/gpgkey'}) 
skipping: [192.168.137.10] => (item={'name': 'nvidia-container-runtime', 'description': 'nvidia-container-runtime', 'baseurl': 'https://nvidia.github.io/nvidia-container-runtime/stable/centos7/$basearch', 'gpgkey': 'https://nvidia.github.io/nvidia-container-runtime/gpgkey'}) 
skipping: [192.168.137.11] => (item={'name': 'libnvidia-container', 'description': 'libnvidia-container', 'baseurl': 'https://nvidia.github.io/libnvidia-container/stable/centos7/$basearch', 'gpgkey': 'https://nvidia.github.io/libnvidia-container/gpgkey'}) 
skipping: [192.168.137.11] => (item={'name': 'nvidia-container-runtime', 'description': 'nvidia-container-runtime', 'baseurl': 'https://nvidia.github.io/nvidia-container-runtime/stable/centos7/$basearch', 'gpgkey': 'https://nvidia.github.io/nvidia-container-runtime/gpgkey'}) 
skipping: [192.168.137.12] => (item={'name': 'libnvidia-container', 'description': 'libnvidia-container', 'baseurl': 'https://nvidia.github.io/libnvidia-container/stable/centos7/$basearch', 'gpgkey': 'https://nvidia.github.io/libnvidia-container/gpgkey'}) 
skipping: [192.168.137.12] => (item={'name': 'nvidia-container-runtime', 'description': 'nvidia-container-runtime', 'baseurl': 'https://nvidia.github.io/nvidia-container-runtime/stable/centos7/$basearch', 'gpgkey': 'https://nvidia.github.io/nvidia-container-runtime/gpgkey'}) 
skipping: [192.168.137.101] => (item={'name': 'libnvidia-container', 'description': 'libnvidia-container', 'baseurl': 'https://nvidia.github.io/libnvidia-container/stable/centos7/$basearch', 'gpgkey': 'https://nvidia.github.io/libnvidia-container/gpgkey'}) 
skipping: [192.168.137.101] => (item={'name': 'nvidia-container-runtime', 'description': 'nvidia-container-runtime', 'baseurl': 'https://nvidia.github.io/nvidia-container-runtime/stable/centos7/$basearch', 'gpgkey': 'https://nvidia.github.io/nvidia-container-runtime/gpgkey'}) 

TASK [containerd : Add nvidia-container-toolkit repository] ****************************************************************************************************************************************************************************************************************************************************************
skipping: [192.168.137.10]
skipping: [192.168.137.11]
skipping: [192.168.137.12]
skipping: [192.168.137.101]

TASK [containerd : Install nvidia-container-runtime] ***********************************************************************************************************************************************************************************************************************************************************************
skipping: [192.168.137.10]
skipping: [192.168.137.11]
skipping: [192.168.137.12]
skipping: [192.168.137.101]

TASK [containerd : Install runc] *******************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.101]
changed: [192.168.137.11]
changed: [192.168.137.10]
changed: [192.168.137.12]

TASK [containerd : Create cni directory] ***********************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]
ok: [192.168.137.101]

TASK [containerd : Install cni] ********************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.12]
changed: [192.168.137.10]
changed: [192.168.137.11]
changed: [192.168.137.101]

TASK [containerd : Install containerd] *************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.11]
changed: [192.168.137.12]
changed: [192.168.137.10]
changed: [192.168.137.101]

TASK [containerd : Set containerd service] *********************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.11]
changed: [192.168.137.12]
changed: [192.168.137.101]

TASK [containerd : Create containerd data directory] ***********************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.11]
changed: [192.168.137.12]
changed: [192.168.137.101]

TASK [containerd : Create containerd config directory] *********************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]
ok: [192.168.137.101]

TASK [containerd : Set containerd config] **********************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.11]
changed: [192.168.137.10]
changed: [192.168.137.101]
changed: [192.168.137.12]

TASK [containerd : Install cri-tools] **************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.12]
changed: [192.168.137.101]
changed: [192.168.137.11]

TASK [containerd : Config cri-tools] ***************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.11]
changed: [192.168.137.12]
changed: [192.168.137.101]

RUNNING HANDLER [containerd : restart containerd] **************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.11]
changed: [192.168.137.12]
changed: [192.168.137.10]
changed: [192.168.137.101]

RUNNING HANDLER [containerd : Containerd | restart containerd] *************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.101]
changed: [192.168.137.11]
changed: [192.168.137.12]
changed: [192.168.137.10]

RUNNING HANDLER [containerd : Containerd | wait for containerd] ************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.11]
changed: [192.168.137.12]
changed: [192.168.137.101]
changed: [192.168.137.10]

RUNNING HANDLER [containerd : Get crictl completion] ***********************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.12]
ok: [192.168.137.101]
ok: [192.168.137.11]

RUNNING HANDLER [containerd : Install crictl completion] *******************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.12]
changed: [192.168.137.101]
changed: [192.168.137.11]

PLAY [Set Worker Nodes] ****************************************************************************************************************************************************************************************************************************************************************************************************

TASK [worker : Install dependency] *****************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]
ok: [192.168.137.101]

TASK [worker : Install dependency] *****************************************************************************************************************************************************************************************************************************************************************************************
skipping: [192.168.137.10]
skipping: [192.168.137.11]
skipping: [192.168.137.12]
skipping: [192.168.137.101]

TASK [worker : Create kubernetes directory] ********************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10] => (item={'line': '/etc/kubernetes/pki'})
ok: [192.168.137.11] => (item={'line': '/etc/kubernetes/pki'})
ok: [192.168.137.12] => (item={'line': '/etc/kubernetes/pki'})
changed: [192.168.137.101] => (item={'line': '/etc/kubernetes/pki'})
changed: [192.168.137.10] => (item={'line': '/etc/kubernetes/manifests'})
changed: [192.168.137.11] => (item={'line': '/etc/kubernetes/manifests'})
changed: [192.168.137.12] => (item={'line': '/etc/kubernetes/manifests'})
changed: [192.168.137.101] => (item={'line': '/etc/kubernetes/manifests'})
ok: [192.168.137.10] => (item={'line': '/var/lib/kubelet'})
ok: [192.168.137.11] => (item={'line': '/var/lib/kubelet'})
ok: [192.168.137.101] => (item={'line': '/var/lib/kubelet'})
ok: [192.168.137.12] => (item={'line': '/var/lib/kubelet'})

TASK [worker : Create kubelet directory] ***********************************************************************************************************************************************************************************************************************************************************************************
skipping: [192.168.137.10]
skipping: [192.168.137.11]
skipping: [192.168.137.12]
skipping: [192.168.137.101]

TASK [worker : Install worker node] ****************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.12] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kubelet'})
changed: [192.168.137.11] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kubelet'})
changed: [192.168.137.10] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kubelet'})
changed: [192.168.137.101] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kubelet'})
changed: [192.168.137.12] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-proxy'})
changed: [192.168.137.11] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-proxy'})
changed: [192.168.137.10] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-proxy'})
changed: [192.168.137.101] => (item={'line': 'http://192.168.224.69:8081/repository/local-files/k8s/v1.26.1/bin/linux/amd64/kube-proxy'})

TASK [worker : Distribution worker certs] **********************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.101] => (item={'line': '/opt/certs/ca.pem'})
changed: [192.168.137.101] => (item={'line': '/opt/certs/ca.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/ca.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/ca.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/ca.pem'})
changed: [192.168.137.101] => (item={'line': '/opt/certs/192.168.137.101/kube-proxy.pem'})
changed: [192.168.137.101] => (item={'line': '/opt/certs/192.168.137.101/kube-proxy.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/ca.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/ca.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/ca.key'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/kube-proxy.pem'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/kube-proxy.pem'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/kube-proxy.pem'})
changed: [192.168.137.10] => (item={'line': '/opt/certs/192.168.137.10/kube-proxy.key'})
changed: [192.168.137.11] => (item={'line': '/opt/certs/192.168.137.11/kube-proxy.key'})
changed: [192.168.137.12] => (item={'line': '/opt/certs/192.168.137.12/kube-proxy.key'})

TASK [worker : Get bootstrap-token-id] *************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.11]
changed: [192.168.137.10]
changed: [192.168.137.12]
changed: [192.168.137.101]

TASK [worker : Get bootstrap-token-secret] *********************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.12]
changed: [192.168.137.101]
changed: [192.168.137.11]

TASK [worker : Distribution worker kubeconfig] *****************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10] => (item={'src': 'bootstrap.kubeconfig.j2', 'dest': '/etc/kubernetes/bootstrap.kubeconfig'})
changed: [192.168.137.11] => (item={'src': 'bootstrap.kubeconfig.j2', 'dest': '/etc/kubernetes/bootstrap.kubeconfig'})
changed: [192.168.137.12] => (item={'src': 'bootstrap.kubeconfig.j2', 'dest': '/etc/kubernetes/bootstrap.kubeconfig'})
changed: [192.168.137.101] => (item={'src': 'bootstrap.kubeconfig.j2', 'dest': '/etc/kubernetes/bootstrap.kubeconfig'})
changed: [192.168.137.10] => (item={'src': 'proxy.kubeconfig.j2', 'dest': '/etc/kubernetes/proxy.kubeconfig'})
changed: [192.168.137.11] => (item={'src': 'proxy.kubeconfig.j2', 'dest': '/etc/kubernetes/proxy.kubeconfig'})
changed: [192.168.137.101] => (item={'src': 'proxy.kubeconfig.j2', 'dest': '/etc/kubernetes/proxy.kubeconfig'})
changed: [192.168.137.12] => (item={'src': 'proxy.kubeconfig.j2', 'dest': '/etc/kubernetes/proxy.kubeconfig'})

TASK [worker : Distribution worker config] *********************************************************************************************************************************************************************************************************************************************************************************
[DEPRECATION WARNING]: Use 'ansible.utils.next_nth_usable' module instead. This feature will be removed from ansible.netcommon in a release after 2024-01-01. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: Use 'ansible.utils.next_nth_usable' module instead. This feature will be removed from ansible.netcommon in a release after 2024-01-01. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: Use 'ansible.utils.next_nth_usable' module instead. This feature will be removed from ansible.netcommon in a release after 2024-01-01. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: Use 'ansible.utils.next_nth_usable' module instead. This feature will be removed from ansible.netcommon in a release after 2024-01-01. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
changed: [192.168.137.10] => (item={'src': 'kubelet.conf.j2', 'dest': '/etc/kubernetes/kubelet.conf'})
changed: [192.168.137.11] => (item={'src': 'kubelet.conf.j2', 'dest': '/etc/kubernetes/kubelet.conf'})
changed: [192.168.137.12] => (item={'src': 'kubelet.conf.j2', 'dest': '/etc/kubernetes/kubelet.conf'})
changed: [192.168.137.101] => (item={'src': 'kubelet.conf.j2', 'dest': '/etc/kubernetes/kubelet.conf'})
changed: [192.168.137.10] => (item={'src': '10-kubelet.conf.j2', 'dest': '/etc/sysconfig/kubelet'})
changed: [192.168.137.11] => (item={'src': '10-kubelet.conf.j2', 'dest': '/etc/sysconfig/kubelet'})
changed: [192.168.137.12] => (item={'src': '10-kubelet.conf.j2', 'dest': '/etc/sysconfig/kubelet'})
changed: [192.168.137.101] => (item={'src': '10-kubelet.conf.j2', 'dest': '/etc/sysconfig/kubelet'})
changed: [192.168.137.10] => (item={'src': 'kube-proxy.conf.j2', 'dest': '/etc/kubernetes/kube-proxy.conf'})
changed: [192.168.137.11] => (item={'src': 'kube-proxy.conf.j2', 'dest': '/etc/kubernetes/kube-proxy.conf'})
changed: [192.168.137.101] => (item={'src': 'kube-proxy.conf.j2', 'dest': '/etc/kubernetes/kube-proxy.conf'})
changed: [192.168.137.12] => (item={'src': 'kube-proxy.conf.j2', 'dest': '/etc/kubernetes/kube-proxy.conf'})

TASK [worker : Distribution worker system unit] ****************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10] => (item={'src': 'kubelet.service.j2', 'dest': '/usr/lib/systemd/system/kubelet.service'})
changed: [192.168.137.11] => (item={'src': 'kubelet.service.j2', 'dest': '/usr/lib/systemd/system/kubelet.service'})
changed: [192.168.137.12] => (item={'src': 'kubelet.service.j2', 'dest': '/usr/lib/systemd/system/kubelet.service'})
changed: [192.168.137.101] => (item={'src': 'kubelet.service.j2', 'dest': '/usr/lib/systemd/system/kubelet.service'})
changed: [192.168.137.10] => (item={'src': 'kube-proxy.service.j2', 'dest': '/usr/lib/systemd/system/kube-proxy.service'})
changed: [192.168.137.11] => (item={'src': 'kube-proxy.service.j2', 'dest': '/usr/lib/systemd/system/kube-proxy.service'})
changed: [192.168.137.12] => (item={'src': 'kube-proxy.service.j2', 'dest': '/usr/lib/systemd/system/kube-proxy.service'})
changed: [192.168.137.101] => (item={'src': 'kube-proxy.service.j2', 'dest': '/usr/lib/systemd/system/kube-proxy.service'})

TASK [worker : Check if bootstrap-token exists] ****************************************************************************************************************************************************************************************************************************************************************************
fatal: [192.168.137.10]: FAILED! => {"changed": true, "cmd": "kubectl -n kube-system get secret bootstrap-token-4febb9", "delta": "0:00:00.712825", "end": "2023-02-07 16:12:57.668711", "msg": "non-zero return code", "rc": 1, "start": "2023-02-07 16:12:56.955886", "stderr": "Error from server (NotFound): secrets \"bootstrap-token-4febb9\" not found", "stderr_lines": ["Error from server (NotFound): secrets \"bootstrap-token-4febb9\" not found"], "stdout": "", "stdout_lines": []}
...ignoring

TASK [worker : Create bootstrap-token secret] ******************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]

TASK [worker : Check if clusterrolebinding kubelet-bootstrap exists] *******************************************************************************************************************************************************************************************************************************************************
fatal: [192.168.137.10]: FAILED! => {"changed": true, "cmd": "kubectl get clusterrolebinding kubelet-bootstrap", "delta": "0:00:00.606823", "end": "2023-02-07 16:13:00.005821", "msg": "non-zero return code", "rc": 1, "start": "2023-02-07 16:12:59.398998", "stderr": "Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io \"kubelet-bootstrap\" not found", "stderr_lines": ["Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io \"kubelet-bootstrap\" not found"], "stdout": "", "stdout_lines": []}
...ignoring

TASK [worker : Create clusterrolebinding kubelet-bootstrap] ****************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]

TASK [worker : Check if node-autoapprove-bootstrap exists] *****************************************************************************************************************************************************************************************************************************************************************
fatal: [192.168.137.10]: FAILED! => {"changed": true, "cmd": "kubectl get clusterrolebinding node-autoapprove-bootstrap", "delta": "0:00:00.609772", "end": "2023-02-07 16:13:02.301013", "msg": "non-zero return code", "rc": 1, "start": "2023-02-07 16:13:01.691241", "stderr": "Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io \"node-autoapprove-bootstrap\" not found", "stderr_lines": ["Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io \"node-autoapprove-bootstrap\" not found"], "stdout": "", "stdout_lines": []}
...ignoring

TASK [worker : Create clusterrolebinding node-autoapprove-bootstrap] *******************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]

TASK [worker : Check if clusterrolebinding node-autoapprove-certificate-rotation exists] ***********************************************************************************************************************************************************************************************************************************
fatal: [192.168.137.10]: FAILED! => {"changed": true, "cmd": "kubectl get clusterrolebinding node-autoapprove-certificate-rotation", "delta": "0:00:00.630910", "end": "2023-02-07 16:13:04.735579", "msg": "non-zero return code", "rc": 1, "start": "2023-02-07 16:13:04.104669", "stderr": "Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io \"node-autoapprove-certificate-rotation\" not found", "stderr_lines": ["Error from server (NotFound): clusterrolebindings.rbac.authorization.k8s.io \"node-autoapprove-certificate-rotation\" not found"], "stdout": "", "stdout_lines": []}
...ignoring

TASK [worker : Create clusterrolebinding node-autoapprove-certificate-rotation] ********************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]

TASK [worker : Restart kubelet] ********************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.101]
changed: [192.168.137.11]
changed: [192.168.137.12]

TASK [worker : Waiting kubelet starting] ***********************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.101]
ok: [192.168.137.12]

TASK [worker : kubelet health check] ***************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.12]
ok: [192.168.137.11]
ok: [192.168.137.10]
ok: [192.168.137.101]

TASK [worker : Restart kube-proxy] *****************************************************************************************************************************************************************************************************************************************************************************************
changed: [192.168.137.10]
changed: [192.168.137.11]
changed: [192.168.137.101]
changed: [192.168.137.12]

TASK [worker : Waiting kube-proxy starting] ********************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.12]
ok: [192.168.137.101]

TASK [worker : kube-proxy health check] ************************************************************************************************************************************************************************************************************************************************************************************
ok: [192.168.137.10]
ok: [192.168.137.11]
ok: [192.168.137.101]
ok: [192.168.137.12]

TASK [worker : Create taint for control-plane] *****************************************************************************************************************************************************************************************************************************************************************************
failed: [192.168.137.10] (item=192.168.137.10) => {"ansible_loop_var": "item", "changed": true, "cmd": "kubectl taint nodes master-01-192.168.137.10 node-role.kubernetes.io/control-plane=:NoSchedule --overwrite", "delta": "0:00:00.610875", "end": "2023-02-07 16:16:03.459949", "item": "192.168.137.10", "msg": "non-zero return code", "rc": 1, "start": "2023-02-07 16:16:02.849074", "stderr": "Error from server (NotFound): nodes \"master-01-192.168.137.10\" not found", "stderr_lines": ["Error from server (NotFound): nodes \"master-01-192.168.137.10\" not found"], "stdout": "", "stdout_lines": []}
failed: [192.168.137.10] (item=192.168.137.11) => {"ansible_loop_var": "item", "changed": true, "cmd": "kubectl taint nodes master-02-192.168.137.11 node-role.kubernetes.io/control-plane=:NoSchedule --overwrite", "delta": "0:00:00.729596", "end": "2023-02-07 16:16:04.576235", "item": "192.168.137.11", "msg": "non-zero return code", "rc": 1, "start": "2023-02-07 16:16:03.846639", "stderr": "Error from server (NotFound): nodes \"master-02-192.168.137.11\" not found", "stderr_lines": ["Error from server (NotFound): nodes \"master-02-192.168.137.11\" not found"], "stdout": "", "stdout_lines": []}
failed: [192.168.137.10] (item=192.168.137.12) => {"ansible_loop_var": "item", "changed": true, "cmd": "kubectl taint nodes master-03-192.168.137.12 node-role.kubernetes.io/control-plane=:NoSchedule --overwrite", "delta": "0:00:00.628364", "end": "2023-02-07 16:16:05.655418", "item": "192.168.137.12", "msg": "non-zero return code", "rc": 1, "start": "2023-02-07 16:16:05.027054", "stderr": "Error from server (NotFound): nodes \"master-03-192.168.137.12\" not found", "stderr_lines": ["Error from server (NotFound): nodes \"master-03-192.168.137.12\" not found"], "stdout": "", "stdout_lines": []}

NO MORE HOSTS LEFT *********************************************************************************************************************************************************************************************************************************************************************************************************

PLAY RECAP *****************************************************************************************************************************************************************************************************************************************************************************************************************
192.168.137.10             : ok=97   changed=44   unreachable=0    failed=1    skipped=12   rescued=0    ignored=4   
192.168.137.101            : ok=59   changed=25   unreachable=0    failed=0    skipped=13   rescued=0    ignored=0   
192.168.137.11             : ok=93   changed=36   unreachable=0    failed=0    skipped=12   rescued=0    ignored=0   
192.168.137.12             : ok=89   changed=36   unreachable=0    failed=0    skipped=12   rescued=0    ignored=0   
localhost                  : ok=47   changed=2    unreachable=0    failed=0    skipped=13   rescued=0    ignored=0 

输出太长了，我截取了一部分。

是否一个大版本打一个tag标签

打了标签后可以指定安装特定版本

希望支持AnoliOS等系统

希望支持AnoliOS等系统

Audit policy metadata-only rule should include `serviceaccounts/token` resource

The following files reference a metadata-only audit policy in order to prevent logging request/response contents for sensitive resources:

• https://github.com/k8sre/k8s/blob/master/roles/master/templates/audit-policy.yaml.j2

A recent Kubernetes bugfix means that audit-logging of subresource requests which previously failed will now log successfully. The serviceaccounts/token subresource responds to TokenRequest API calls with a newly minted service account token.

The serviceaccounts/token resource should also be included in the metadata-only audit policy if credentials are not intended to appear in the audit log:

- group: "" # core
  resources: ["secrets", "configmaps", "serviceaccounts/token"]

大佬有更新ansible吗？现在这个版本执行起来报错太多了

比如：
1、
fatal: [localhost]: FAILED! => {"msg": "template error while templating string: no filter named 'next_nth_usable'. String: IP:{{ kubernetes.serviceSubnet | next_nth_usable(1) }}"}

2、
fatal: [10.1.1.193]: FAILED! => {"msg": "The conditional check 'kubernetes.podSubnet | ipaddr(kubernetes.serviceSubnet) | string == 'None'' failed. The error was: template error while templating string: no filter named 'ipaddr'. String: {% if kubernetes.podSubnet | ipaddr(kubernetes.serviceSubnet) | string == 'None' %} True {% else %} False {% endif %}"}

3、
TASK [verify : Check that podSubnet is a network range] **************************************************************************************************************************
fatal: [10.1.1.193]: FAILED! => {"msg": "The conditional check 'kubernetes.podSubnet | ipaddr('net')' failed. The error was: template error while templating string: no filter named 'ipaddr'. String: {% if kubernetes.podSubnet | ipaddr('net') %} True {% else %} False {% endif %}"}

global link for url and mirrors

Hi,

Can you help with the global urls and mirrors and registry rather it been going towards aliyuncs.com, https://k8s-gcr-io.mirrors.sjtug.sjtu.edu.cn, registry.aliyuncs.com, docker.mirrors.sjtug.sjtu.edu.cn its not accessible for me.

help or guidance would be very helpful

offline packages

need to prepare the relevant offline packages by myself?