Drain Node Error

[dstrebel]

Drain Node Error

[Lawouach]

Hi,

Sorry for being so long to respond. The API had indeed changed and it's been fixed in master but it's not released it yet. I should be releasing this week a new version of that's fine?

[Lawouach]

This has now been released. Could you please try and let us know if it solved the pb?

[nitinbhat]

Draining a node is failing.

[2024-05-06 06:53:11 DEBUG] [actions:75] Found 0 nodes
[2024-05-06 06:53:11 DEBUG] [activity:317] Activity failed
Traceback (most recent call last):
File "/home/dev/.venvs/chaostk/lib/python3.9/site-packages/chaoslib/provider/python.py", line 56, in run_python_activity
return func(**arguments)
File "/home/dev/.venvs/chaostk/lib/python3.9/site-packages/chaosk8s/node/actions.py", line 307, in drain_nodes
nodes = _select_nodes(
File "/home/dev/.venvs/chaostk/lib/python3.9/site-packages/chaosk8s/node/actions.py", line 94, in _select_nodes
raise ActivityFailed("failed to find a node that matches selector")
chaoslib.exceptions.ActivityFailed: failed to find a node that matches selector

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dev/.venvs/chaostk/lib/python3.9/site-packages/chaoslib/activity.py", line 310, in run_activity
    result = run_python_activity(activity, configuration, secrets)
  File "/home/dev/.venvs/chaostk/lib/python3.9/site-packages/chaoslib/provider/python.py", line 58, in run_python_activity
    raise ActivityFailed(
  File "/home/dev/.venvs/chaostk/lib/python3.9/site-packages/chaoslib/provider/python.py", line 56, in run_python_activity
    return func(**arguments)
  File "/home/dev/.venvs/chaostk/lib/python3.9/site-packages/chaosk8s/node/actions.py", line 307, in drain_nodes
    nodes = _select_nodes(
  File "/home/dev/.venvs/chaostk/lib/python3.9/site-packages/chaosk8s/node/actions.py", line 94, in _select_nodes
    raise ActivityFailed("failed to find a node that matches selector")
chaoslib.exceptions.ActivityFailed: chaoslib.exceptions.ActivityFailed: failed to find a node that matches selector

[2024-05-06 06:53:11 ERROR] [activity:268] => failed: chaoslib.exceptions.ActivityFailed: failed to find a node that matches selector

This is valid label though. Am I missing anything here?

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME LABELS datkube-control-plane Ready control-plane 178m v1.26.3 10.101.0.2 <none> Ubuntu 22.04.2 LTS 5.10.0-22-amd64 containerd://1.6.19-46-g941215f49 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=datkube-control-plane,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers= datkube-worker Ready <none> 177m v1.26.3 10.101.0.5 <none> Ubuntu 22.04.2 LTS 5.10.0-22-amd64 containerd://1.6.19-46-g941215f49 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=datkube-worker,kubernetes.io/os=linux,tmm=true datkube-worker2 Ready <none> 177m v1.26.3 10.101.0.4 <none> Ubuntu 22.04.2 LTS 5.10.0-22-amd64 containerd://1.6.19-46-g941215f49 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=datkube-worker2,kubernetes.io/os=linux,tmm=true datkube-worker3 Ready <none> 177m v1.26.3 10.101.0.3 <none> Ubuntu 22.04.2 LTS 5.10.0-22-amd64 containerd://1.6.19-46-g941215f49 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=datkube-worker3,kubernetes.io/os=linux,tmm=true

This is issue is also seen with all node operations like cordon_node, delete_node etc.

[Lawouach]

What was your selector?

[nitinbhat]

What was your selector?

label_selector: "kubernetes.io/hostname=datkube-worker3"

$kgn --show-labels | grep worker3
datkube-worker3 Ready 5h55m v1.26.3 10.101.0.3 Ubuntu 22.04.2 LTS 5.10.0-22-amd64 containerd://1.6.19-46-g941215f49 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=datkube-worker3,kubernetes.io/os=linux,tmm=true

[Lawouach]

Thanks. Can you show me the entire action as you used it please?

[nitinbhat]

Thanks. Can you show me the entire action as you used it please?

Sure

• type: action
  name: drain-node
  provider:
  type: python
  module: chaosk8s.node.actions
  func: drain_nodes
  arguments:
  name: drain-nodes
  label_selector: "kubernetes.io/hostname=datkube-worker3"

[Lawouach]

Thanks. I think it's because you set name: drain-nodes in the arguments. Your node is called datkube-control-plane. But I would ignore the name argument entirely if you don't want to aim a particular node with the given label.

[nitinbhat]

Thanks. After correcting name, this works as expected.

[Lawouach]

Fantastic news! :)