chaostoolkit/chaostoolkit-kubernetes

Kubernetes driver extension of the Chaos Toolkit probes and actions API

chaostoolkit-kubernetes's Issues

Cleaning and rework

The discussion started here. This issue is here to share thoughts and plan a small rework to clean up the repository.

The plugin is actually a bit messy because of the background. The most annoying things (making it hard for a user to understand what to use and when) is that, at the beginning of the project, some actions and probes meant to be high-level. This is also why the project still contains unnamespaced actions and probes. Also, as I mentioned in the PR linked above, some functions have the same prefix but don't work with the same resources (cf. microservice_available_and_healthy function which looks for a deployment vs. microservice_is_not_available which looks for pods). Finally, some of them may be a bit redundant.

As you can see, the actions and probes included recently are namespaced per resource (pod and node actually).

Here are two things I think we should do for now:

• namespace the high-level functions to be consistent with the newly added ones
• rename the high-level functions to have a clear name of the resource they are working with (for instance, do not use microservice_ prefix anymore since it does not relate to a Kubernetes resource)
• remove/deprecate redundant functions and opt it for more flexible ones to cover more use cases and keep it clear

We should still keep backward compatibility with existing functions and mark them as deprecated to remove them in a next release.

list_pod_for_all_namespaces no longer takes the include_uninitialized boolean parameter

Suggest removing it.

Include grace_period_seconds to the terminate_pods action

It would be interesting to add grace_period_seconds paramater to the V1DeleteOptions at chaosk8s/pod/actions.py so we can choose whether we want to "kill" a pod (terminate it as soon as possible, i.e. grace_period_seconds=0) instead of gracefully terminating it.

kill_microservice not working due to naked 'body' args for functions

This looks similar to #34

It appears actions.py has several incorrect instances of body as an argument.

If I edit to read body=body as in here for kill_microservice: https://github.com/chaostoolkit/chaostoolkit-kubernetes/blob/master/chaosk8s/actions.py#L67

The deployment will correctly be deleted. Without that edit, I will get the following:

[2019-06-19 17:32:43 ERROR]   => failed: TypeError: delete_namespaced_deployment() takes 3 positional arguments but 4 were given

And the function will not run, no deployment is deleted.

This also appears two other places:

https://github.com/chaostoolkit/chaostoolkit-kubernetes/blob/master/chaosk8s/actions.py#L81

https://github.com/chaostoolkit/chaostoolkit-kubernetes/blob/master/chaosk8s/actions.py#L95

chaos discover invalid syntax

$ chaos discover chaostoolkit-kubernetes --no-install
Traceback (most recent call last):
File "/usr/bin/chaos", line 6, in
from chaostoolkit.main import cli
File "/usr/lib/python2.7/site-packages/chaostoolkit/main.py", line 20
def run(report_path: str, dry: bool, path: str):
^
SyntaxError: invalid syntax

remove_service_endpoint missing argument request

When I run the experiment to dremove service end point I receive the error below:
"TypeError: delete_namespaced_service() missing 1 required positional argument: 'body'\n",
when I check the actions code no "body" argument requested for the delete_namespaced_service

https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/CoreV1Api.md#delete_namespaced_service

Support for statefullsets

I would like to use chaostoolkit to test statefullsets the same way that it is possible to test deployments in kubernetes.

TypeError: delete_namespaced_pod() takes 3 positional arguments but 4 were given

I am getting this error:

TypeError: delete_namespaced_pod() takes 3 positional arguments but 4 were given
when trying to kill a pod with latest chaostolkit-kubernetes ..

here is the json:

{
    "version": "1.0.0",
    "title": "System is resilient to provider's failures",
    "description": "Can our consumer survive gracefully a provider's failure?",
    "method": [
        {
            "type": "probe",
            "name": "count-pods-before-termination",
            "provider": {
                "type": "python",
                "module": "chaosk8s.pod.probes",
                "func": "count_pods",
                "arguments": {
                    "label_selector": "app=nginx",
                    "ns": "ns1"
                }
            },
            "pauses": {
                "after": 10
            }
        },
        {
            "type": "action",
            "name": "Terminate_pod",
            "provider": {
                "type": "python",
                "module": "chaosk8s.pod.actions",
                "func": "terminate_pods",
                "arguments": {
                    "label_selector": "app=nginx",
                    "ns": "ns1"
                }
            },
            "pauses": {
                "after": 2
            }
        },
        {
            "type": "probe",
            "name": "count-pods-after-termintation",
            "provider": {
                "type": "python",
                "module": "chaosk8s.pod.probes",
                "func": "count_pods",
                "arguments": {
                    "label_selector": "app=nginx",
                    "ns": "ns1"
                }
            },
            "pauses": {
                "after": 10
            }
        }

    ],
    "rollbacks": [
    ]
}

gitignore .idea folder

Is it possible to ignore .idea folder ?
I use Intellij and i need to ignore thoses files every times i create a branch on chaostoolkit-kubernetes project

Do i need to create a pull request for that ?

CHAOSTOOLKIT_IN_POD env variable not working in kubernetes

Looks like exception is thrown when using the CHAOSTOOLKIT_IN_POD environment variable in the kubernetes:

stack trace from the logs:

Traceback (most recent call last):
  File "/usr/local/bin/chaos", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/chaostoolkit/cli.py", line 106, in run
    experiment = load_experiment(
  File "/usr/local/lib/python3.8/site-packages/chaoslib/loader.py", line 96, in load_experiment
    parsed = parse_experiment_from_file(experiment_source)
  File "/usr/local/lib/python3.8/site-packages/chaoslib/loader.py", line 38, in parse_experiment_from_file
    return json.load(f)
  File "/usr/local/lib/python3.8/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/usr/local/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.8/json/decoder.py", line 353, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 8 column 3 (char 213)

New Release

The changelog mentions a few unreleased features. @Lawouach when can we expect a new release?

random pod deletion doesn't capture targeted pod details in journal file

The journal.json file generated as a result of random pod deletion experiment is missing details about the targeted pod.

Add developer installation instructions to README

What is the tolerance for read_pod_logs

the following probe fails with error:

Steady state probe 'fetch-ziptranslatorservice-zipservice-pod-logs' is not in the given tolerance so failing this experiment

{
"name": "fetch-ziptranslatorservice-zipservice-pod-logs",
"type": "probe",
"tolerance": true,
"provider": {
"func": "read_pod_logs",
"arguments": {
"label_selector": "name=zipservice",
"last": "30m",
"ns": "core"
},
"type": "python",
"module": "chaosk8s.pod.probes"
}
}

Correct typos in README

Problem: Kubernetes local file configuration does not set the proxy

Using the option to connect to the kubernetes cluster via local configuation file does not set a proxy. I have a working example below:

import os
from kubernetes import client, config
from kubernetes.client import Configuration

# Configs can be set in Configuration class directly or using helper utility
proxy_url = os.getenv('HTTP_PROXY', None)
config.load_kube_config()
if proxy_url:
    client.Configuration._default.proxy = proxy_url

v1 = client.CoreV1Api()
print("Listing pods with their IPs:")
ret = v1.list_pod_for_all_namespaces(watch=False)
for i in ret.items:
    print("%s\t%s\t%s" % (i.status.pod_ip, i.metadata.namespace, i.metadata.name))

Failed to discover chaostoolkit-kubernetes

First, I've successfully installed chaostoolkit and chaostoolkit-kubernetes from pip in my venv :

⇒  python3 -m venv chaostk
⇒  source chaostk/bin/activate
⇒  pip install chaostoolkit
⇒  pip install chaostoolkit-kubernetes

I am trying to use chaostoolkit-kubernetes on my k8s cluster, but I always get the following errors :

⇒  which chaos
~/venv/chaostk/bin/chaos
⇒  chaos --verbose discover chaostoolkit-kubernetes
[2020-10-28 11:05:33 DEBUG] [cli:74] ###############################################################################
[2020-10-28 11:05:33 DEBUG] [cli:75] Running command 'discover'
[2020-10-28 11:05:33 DEBUG] [cli:79] Using settings file '/home/xat/.chaostoolkit/settings.yaml'
[2020-10-28 11:05:33 DEBUG] [settings:26] The Chaos Toolkit settings file could not be found at '/home/xat/.chaostoolkit/settings.yaml'.
[2020-10-28 11:05:33 INFO] [package:22] Attempting to download and install package 'chaostoolkit-kubernetes'
[2020-10-28 11:05:34 DEBUG] [package:31] Requirement already up-to-date: chaostoolkit-kubernetes in /home/xat/venv/chaostk/lib/python3.8/site-packages (0.24.0)
    Requirement already satisfied, skipping upgrade: pyyaml in /home/xat/venv/chaostk/lib/python3.8/site-packages (from chaostoolkit-kubernetes) (5.3.1)
    Requirement already satisfied, skipping upgrade: chaostoolkit-lib>=0.20.0 in /home/xat/venv/chaostk/lib/python3.8/site-packages (from chaostoolkit-kubernetes) (1.15.0)
    Requirement already satisfied, skipping upgrade: dateparser in /home/xat/venv/chaostk/lib/python3.8/site-packages (from chaostoolkit-kubernetes) (0.7.6)
    Requirement already satisfied, skipping upgrade: kubernetes in /home/xat/venv/chaostk/lib/python3.8/site-packages (from chaostoolkit-kubernetes) (12.0.0)
    Requirement already satisfied, skipping upgrade: logzero in /home/xat/venv/chaostk/lib/python3.8/site-packages (from chaostoolkit-kubernetes) (1.5.0)
    Requirement already satisfied, skipping upgrade: requests>=2.21 in /home/xat/venv/chaostk/lib/python3.8/site-packages (from chaostoolkit-lib>=0.20.0->chaostoolkit-kubernetes) (2.24.0)
    Requirement already satisfied, skipping upgrade: tzlocal in /home/xat/venv/chaostk/lib/python3.8/site-packages (from dateparser->chaostoolkit-kubernetes) (2.1)
    Requirement already satisfied, skipping upgrade: regex!=2019.02.19 in /home/xat/venv/chaostk/lib/python3.8/site-packages (from dateparser->chaostoolkit-kubernetes) (2020.10.23)
    Requirement already satisfied, skipping upgrade: pytz in /home/xat/venv/chaostk/lib/python3.8/site-packages (from dateparser->chaostoolkit-kubernetes) (2020.1)
    Requirement already satisfied, skipping upgrade: python-dateutil in /home/xat/venv/chaostk/lib/python3.8/site-packages (from dateparser->chaostoolkit-kubernetes) (2.8.1)
    Requirement already satisfied, skipping upgrade: certifi>=14.05.14 in /home/xat/venv/chaostk/lib/python3.8/site-packages (from kubernetes->chaostoolkit-kubernetes) (2020.6.20)
    Requirement already satisfied, skipping upgrade: google-auth>=1.0.1 in /home/xat/venv/chaostk/lib/python3.8/site-packages (from kubernetes->chaostoolkit-kubernetes) (1.22.1)
    Requirement already satisfied, skipping upgrade: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /home/xat/venv/chaostk/lib/python3.8/site-packages (from kubernetes->chaostoolkit-kubernetes) (0.57.0)
    Requirement already satisfied, skipping upgrade: requests-oauthlib in /home/xat/venv/chaostk/lib/python3.8/site-packages (from kubernetes->chaostoolkit-kubernetes) (1.3.0)
    Requirement already satisfied, skipping upgrade: setuptools>=21.0.0 in /home/xat/venv/chaostk/lib/python3.8/site-packages (from kubernetes->chaostoolkit-kubernetes) (41.6.0)
    Requirement already satisfied, skipping upgrade: urllib3>=1.24.2 in /home/xat/venv/chaostk/lib/python3.8/site-packages (from kubernetes->chaostoolkit-kubernetes) (1.25.11)
    Requirement already satisfied, skipping upgrade: six>=1.9.0 in /home/xat/venv/chaostk/lib/python3.8/site-packages (from kubernetes->chaostoolkit-kubernetes) (1.15.0)
    Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /home/xat/venv/chaostk/lib/python3.8/site-packages (from requests>=2.21->chaostoolkit-lib>=0.20.0->chaostoolkit-kubernetes) (2.10)
    Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in /home/xat/venv/chaostk/lib/python3.8/site-packages (from requests>=2.21->chaostoolkit-lib>=0.20.0->chaostoolkit-kubernetes) (3.0.4)
    Requirement already satisfied, skipping upgrade: pyasn1-modules>=0.2.1 in /home/xat/venv/chaostk/lib/python3.8/site-packages (from google-auth>=1.0.1->kubernetes->chaostoolkit-kubernetes) (0.2.8)
    Requirement already satisfied, skipping upgrade: cachetools<5.0,>=2.0.0 in /home/xat/venv/chaostk/lib/python3.8/site-packages (from google-auth>=1.0.1->kubernetes->chaostoolkit-kubernetes) (4.1.1)
    Requirement already satisfied, skipping upgrade: rsa<5,>=3.1.4; python_version >= "3.5" in /home/xat/venv/chaostk/lib/python3.8/site-packages (from google-auth>=1.0.1->kubernetes->chaostoolkit-kubernetes) (4.6)
    Requirement already satisfied, skipping upgrade: oauthlib>=3.0.0 in /home/xat/venv/chaostk/lib/python3.8/site-packages (from requests-oauthlib->kubernetes->chaostoolkit-kubernetes) (3.1.0)
    Requirement already satisfied, skipping upgrade: pyasn1<0.5.0,>=0.4.6 in /home/xat/venv/chaostk/lib/python3.8/site-packages (from pyasn1-modules>=0.2.1->google-auth>=1.0.1->kubernetes->chaostoolkit-kubernetes) (0.4.8)
    
[2020-10-28 11:05:34 INFO] [package:40] Package downloaded and installed in current environment
[2020-10-28 11:05:34 DEBUG] [cli:444] Failed to discover chaostoolkit-kubernetes
    Traceback (most recent call last):
      File "/home/xat/.local/lib/python3.8/site-packages/chaoslib/discovery/package.py", line 99, in get_importname_from_package
        dist = importlib_metadata.distribution(package_name)
      File "/usr/lib64/python3.8/importlib/metadata.py", line 504, in distribution
        return Distribution.from_name(distribution_name)
      File "/usr/lib64/python3.8/importlib/metadata.py", line 177, in from_name
        raise PackageNotFoundError(name)
    importlib.metadata.PackageNotFoundError: chaostoolkit-kubernetes
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "/home/xat/.local/lib/python3.8/site-packages/chaostoolkit/cli.py", line 439, in discover
        discovery = disco(
      File "/home/xat/.local/lib/python3.8/site-packages/chaoslib/discovery/discover.py", line 33, in discover
        package = load_package(package_name)
      File "/home/xat/.local/lib/python3.8/site-packages/chaoslib/discovery/package.py", line 47, in load_package
        name = get_importname_from_package(package_name)
      File "/home/xat/.local/lib/python3.8/site-packages/chaoslib/discovery/package.py", line 101, in get_importname_from_package
        raise DiscoveryFailed(
    chaoslib.exceptions.DiscoveryFailed: Package chaostoolkit-kubernetes not found 
[2020-10-28 11:05:34 CRITICAL] [cli:445] Package chaostoolkit-kubernetes not found 

I also tried to run chaos discover chaostoolkit-aws that works well.

Thanks for the support.

What do you mean by "name" parameter in function microservice_available_and_healthy?

In the documentation it states look for deployment by "name" but when I give the deployment name it always fails stating microservice not found.

Provide a new function to remove a service

A new function to remove k8s service would be useful:

"provider": {
                "type": "python",
                "module": "chaosk8s.service.actions",
                "func": "remove_service",
                "arguments": {
                    "label_selector": "app=myapp-greeting",
                    "ns": "default"
                }
            }

FYI @adrianco

Repo description says "Chas toolkit"

Sorry, OCD. Can't PR that.

[OCP v3.7.42] [kubernetes v1.7.6+a08f5eeb62] [action] [scale_microservice]

openshift v3.7.42
kubernetes v1.7.6+a08f5eeb62
chaos, version 0.14.0

[2018-06-22 10:53:57 INFO] [activity:158] Action: scale_microservice
[2018-06-22 10:53:57 DEBUG] [activity:225] Activity failed
Reason: Internal Server Error
HTTP response headers: HTTPHeaderDict({'Content-Length': '117', 'Date': 'Fri, 22 Jun 2018 10:53:57 GMT', 'Content-Type': 'application/json', 'Cache-Control': 'no-store'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"unrecognized type: int32","code":500}

chaosToolKit.log

AKS restart node

{
"name": "restart-node",
"type": "action",
"provider": {
"func": "restart_node",
"type": "python",
"module": "chaosazure.aks.actions"
}
}

Will the above action restart all the nodes in the given AKS?

Add discovery function

setup.py summary does not follow CTK extension format

in the CTK cli, the extension summary is supposed to start with the prefix "Chaos Toolkit Extension for "

chaostoolkit-kubernetes/setup.py

Line 25 in b5de1aa

desc = 'Chaos Toolkit Kubernetes support'

base64 padding in jwt tokens

Hi
Im chaostoolkit-lib 0.21.0. Python 3.6.0.
Last week we changed our auth method in kubernetes. We use jwt tokens now.
There is known issue for python and oidc tokens

kubernetes-client/python#525

For now ive disabled using of kube configuration and authenticate with plain token only.
But this situation have to be addressed somehow.

Drain Nodes/Uncordon Nodes does not work properly

the label_selector/name does not work, is this a similar issue like the one we discussed yesterday. I have also tries adding a label selector with "name" tag

Is iterative action possible?

Is it possible to have multiple iteration of single action in an experiment?
Say I want to run terminate-db-pod several times after each 600 sec.

{ "name": "all-our-microservices-should-be-healthy", "type": "probe", "tolerance": "true", "provider": { "type": "python", "module": "chaosk8s.probes", "func": "microservice_available_and_healthy", "arguments": { "name": "myapp", "ns": "myns" } } }, { "type": "action", "name": "terminate-db-pod", "provider": { "type": "python", "module": "chaosk8s.pod.actions", "func": "terminate_pods", "arguments": { "label_selector": "app=my-app", "name_pattern": "my-app-[0-9]$", "rand": true, "ns": "default" } }, "pauses": { "after": 5 } }

Windows Doc Update - Hitting from outside GCP Cluster

Hi,

Feel free to reject or accept my read me doc update recommendation.

if you are hitting from outside of cluster. download kubernetes config file and place in your home folder in the below format in Windows -> C:\Users\username\.kube\config, Chaos toolkit automatically picks up from this folder

More Detailed Notes for GOOGLE_APPLICATION_CREDENTIALS

Create Google Cloud Service Account and download the json file and place it in the following format for google cloud.

open command prompt in windows and enter the following command
set GOOGLE_APPLICATION_CREDENTIALS="C:\Users\username\micro-primer-####.json

Add Port-forward action

I want an action to activate port-forwarding to a service or pod during the experiment. This will allow me to hit private endpoints that are not exposed. Currently, I manually set up the port forwarding using kubectl before the experiment begins. https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/
I would like to add it as an action.

Count pods probe is inconsistent.

Hi, we have been trying the probe count pods and faced inconsistency.
When we give a range as tolerance it is increasing the pod count and also the return of pod count is incorrect.
Another point phase check is also inconsistent, it does not recognise the running pods and prompting as pending even if they are running state when its checked with kubectl.

�[36m[2019-02-01 14:49:52 DEBUG]�[39m The Chaos Toolkit settings file could not be found at '/home/jakob/.chaostoolkit/settings.yaml'.
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Building activity cache...
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Cached 2 activities
�[32m[2019-02-01 14:49:52 INFO]�[39m Validating the experiment's syntax
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Loading configuration...
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Loading secrets...
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Secrets loaded
�[32m[2019-02-01 14:49:52 INFO]�[39m Experiment looks valid
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Clearing activities cache
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Building activity cache...
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Cached 2 activities
�[32m[2019-02-01 14:49:52 INFO]�[39m Running experiment: Test janie-nginx Resilience - at least one pod
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Loading configuration...
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Loading secrets...
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Secrets loaded
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Initializing controls
�[32m[2019-02-01 14:49:52 INFO]�[39m Steady state hypothesis: Prometheus running as expected
�[32m[2019-02-01 14:49:52 INFO]�[39m Probe: count_pods
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Activity 'count_pods' loaded from '/usr/lib/python3.7/site-packages/chaosk8s/pod/probes.py'
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Using Kubernetes context: default
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Found 2 pods matching label 'app=janie-nginx' in ns 'chaos'
�[36m[2019-02-01 14:49:52 DEBUG]�[39m => succeeded with '2'
�[36m[2019-02-01 14:49:52 DEBUG]�[39m allowed tolerance is [1, 3]
�[32m[2019-02-01 14:49:52 INFO]�[39m Steady state hypothesis is met!
�[32m[2019-02-01 14:49:52 INFO]�[39m Action: terminate_pods
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Activity 'terminate_pods' loaded from '/usr/lib/python3.7/site-packages/chaosk8s/pod/actions.py'
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Using Kubernetes context: default
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Found 2 pods labelled 'app=janie-nginx' in ns chaos
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Pod 'janie-nginx-5795fbf867-l6l4b' match pattern
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Pod 'janie-nginx-5795fbf867-vcv2p' match pattern
�[36m[2019-02-01 14:49:52 DEBUG]�[39m Picked pods 'janie-nginx-5795fbf867-l6l4b,janie-nginx-5795fbf867-vcv2p' to be terminated
�[36m[2019-02-01 14:49:52 DEBUG]�[39m => succeeded without any result value
�[32m[2019-02-01 14:49:52 INFO]�[39m Pausing after activity for 5s...
�[32m[2019-02-01 14:49:57 INFO]�[39m Steady state hypothesis: Prometheus running as expected
�[32m[2019-02-01 14:49:57 INFO]�[39m Probe: count_pods
�[36m[2019-02-01 14:49:57 DEBUG]�[39m Activity 'count_pods' loaded from '/usr/lib/python3.7/site-packages/chaosk8s/pod/probes.py'
�[36m[2019-02-01 14:49:57 DEBUG]�[39m Using Kubernetes context: default
�[36m[2019-02-01 14:49:58 DEBUG]�[39m Found 2 pods matching label 'app=janie-nginx' in ns 'chaos'
�[36m[2019-02-01 14:49:58 DEBUG]�[39m => succeeded with '2'
�[36m[2019-02-01 14:49:58 DEBUG]�[39m allowed tolerance is [1, 3]
�[32m[2019-02-01 14:49:58 INFO]�[39m Steady state hypothesis is met!
�[32m[2019-02-01 14:49:58 INFO]�[39m Let's rollback...
�[32m[2019-02-01 14:49:58 INFO]�[39m No declared rollbacks, let's move on.
�[32m[2019-02-01 14:49:58 INFO]�[39m Experiment ended with status: completed
�[36m[2019-02-01 14:49:58 DEBUG]�[39m Cleaning up controls
�[36m[2019-02-01 14:49:58 DEBUG]�[39m Clearing activities cache
�[36m[2019-02-01 14:51:00 DEBUG]�[39m ###############################################################################
�[36m[2019-02-01 14:51:00 DEBUG]�[39m Running command 'run'
�[36m[2019-02-01 14:51:00 DEBUG]�[39m Using settings file '/home/jakob/.chaostoolkit/settings.yaml'
�[33m[2019-02-01 14:51:01 WARNING]�[39m
There is a new version (1.0.0rc3) of the chaostoolkit available.
You may upgrade by typing:

$ pip install -U chaostoolkit

Please review changes at https://github.com/chaostoolkit/chaostoolkit/blob/master/CHANGELOG.md

�[36m[2019-02-01 14:51:01 DEBUG]�[39m The Chaos Toolkit settings file could not be found at '/home/jakob/.chaostoolkit/settings.yaml'.
�[36m[2019-02-01 14:51:01 DEBUG]�[39m Building activity cache...
�[36m[2019-02-01 14:51:01 DEBUG]�[39m Cached 2 activities
�[32m[2019-02-01 14:51:01 INFO]�[39m Validating the experiment's syntax
�[36m[2019-02-01 14:51:01 DEBUG]�[39m Loading configuration...
�[36m[2019-02-01 14:51:01 DEBUG]�[39m Loading secrets...
�[36m[2019-02-01 14:51:01 DEBUG]�[39m Secrets loaded
�[32m[2019-02-01 14:51:01 INFO]�[39m Experiment looks valid
�[36m[2019-02-01 14:51:01 DEBUG]�[39m Clearing activities cache
�[36m[2019-02-01 14:51:01 DEBUG]�[39m Building activity cache...
�[36m[2019-02-01 14:51:01 DEBUG]�[39m Cached 2 activities
�[32m[2019-02-01 14:51:01 INFO]�[39m Running experiment: Test janie-nginx Resilience - at least one pod
�[36m[2019-02-01 14:51:01 DEBUG]�[39m Loading configuration...
�[36m[2019-02-01 14:51:01 DEBUG]�[39m Loading secrets...
�[36m[2019-02-01 14:51:01 DEBUG]�[39m Secrets loaded
�[36m[2019-02-01 14:51:01 DEBUG]�[39m Initializing controls
�[32m[2019-02-01 14:51:01 INFO]�[39m Steady state hypothesis: Prometheus running as expected
�[32m[2019-02-01 14:51:01 INFO]�[39m Probe: count_pods
�[36m[2019-02-01 14:51:01 DEBUG]�[39m Activity 'count_pods' loaded from '/usr/lib/python3.7/site-packages/chaosk8s/pod/probes.py'
�[36m[2019-02-01 14:51:01 DEBUG]�[39m Using Kubernetes context: default
�[36m[2019-02-01 14:51:03 DEBUG]�[39m Found 2 pods matching label 'app=janie-nginx' in ns 'chaos'
�[36m[2019-02-01 14:51:03 DEBUG]�[39m => succeeded with '2'
�[36m[2019-02-01 14:51:03 DEBUG]�[39m allowed tolerance is [1, 2]
�[32m[2019-02-01 14:51:03 INFO]�[39m Steady state hypothesis is met!
�[32m[2019-02-01 14:51:03 INFO]�[39m Action: terminate_pods
�[36m[2019-02-01 14:51:03 DEBUG]�[39m Activity 'terminate_pods' loaded from '/usr/lib/python3.7/site-packages/chaosk8s/pod/actions.py'
�[36m[2019-02-01 14:51:03 DEBUG]�[39m Using Kubernetes context: default
�[36m[2019-02-01 14:51:03 DEBUG]�[39m Found 2 pods labelled 'app=janie-nginx' in ns chaos
�[36m[2019-02-01 14:51:03 DEBUG]�[39m Pod 'janie-nginx-5795fbf867-7wjdt' match pattern
�[36m[2019-02-01 14:51:03 DEBUG]�[39m Pod 'janie-nginx-5795fbf867-zkbvf' match pattern
�[36m[2019-02-01 14:51:03 DEBUG]�[39m Picked pods 'janie-nginx-5795fbf867-7wjdt,janie-nginx-5795fbf867-zkbvf' to be terminated
�[36m[2019-02-01 14:51:03 DEBUG]�[39m => succeeded without any result value
�[32m[2019-02-01 14:51:03 INFO]�[39m Pausing after activity for 10s...
�[32m[2019-02-01 14:51:13 INFO]�[39m Steady state hypothesis: Prometheus running as expected
�[32m[2019-02-01 14:51:13 INFO]�[39m Probe: count_pods
�[36m[2019-02-01 14:51:13 DEBUG]�[39m Activity 'count_pods' loaded from '/usr/lib/python3.7/site-packages/chaosk8s/pod/probes.py'
�[36m[2019-02-01 14:51:13 DEBUG]�[39m Using Kubernetes context: default
�[36m[2019-02-01 14:51:14 DEBUG]�[39m Found 4 pods matching label 'app=janie-nginx' in ns 'chaos'
�[36m[2019-02-01 14:51:14 DEBUG]�[39m => succeeded with '4'
�[36m[2019-02-01 14:51:14 DEBUG]�[39m allowed tolerance is [1, 2]
[2019-02-01 14:51:14 CRITICAL] Steady state probe 'count_pods' is not in the given tolerance so failing this experiment
�[32m[2019-02-01 14:51:14 INFO]�[39m Let's rollback...
�[32m[2019-02-01 14:51:14 INFO]�[39m No declared rollbacks, let's move on.
�[32m[2019-02-01 14:51:14 INFO]�[39m Experiment ended with status: deviated
�[32m[2019-02-01 14:51:14 INFO]�[39m The steady-state has deviated, a weakness may have been discovered
�[36m[2019-02-01 14:51:14 DEBUG]�[39m Cleaning up controls
�[36m[2019-02-01 14:51:14 DEBUG]�[39m Clearing activities cache
�[36m[2019-02-01 14:52:21 DEBUG]�[39m ###############################################################################
�[36m[2019-02-01 14:52:21 DEBUG]�[39m Running command 'run'
�[36m[2019-02-01 14:52:21 DEBUG]�[39m Using settings file '/home/jakob/.chaostoolkit/settings.yaml'
�[33m[2019-02-01 14:52:22 WARNING]�[39m
There is a new version (1.0.0rc3) of the chaostoolkit available.
You may upgrade by typing:

---------------____________________________------------------------------_____________________________
�[36m[2019-02-01 14:31:47 DEBUG]�[39m Activity 'pods_in_phase' loaded from '/usr/lib/python3.7/site-packages/chaosk8s/pod/probes.py'
�[36m[2019-02-01 14:31:47 DEBUG]�[39m Using Kubernetes context: default
�[36m[2019-02-01 14:31:47 DEBUG]�[39m Found 4 pods matching label 'app=janie-nginx' in ns 'chaos'
�[36m[2019-02-01 14:31:47 DEBUG]�[39m Activity failed
Traceback (most recent call last):
File "/usr/lib/python3.7/site-packages/chaoslib/provider/python.py", line 57, in run_python_activity
return func(**arguments)
File "/usr/lib/python3.7/site-packages/chaosk8s/pod/probes.py", line 105, in pods_in_phase
name=label_selector, s=d.status.phase, p=phase))
chaoslib.exceptions.ActivityFailed: pod 'app=janie-nginx' is in phase 'Pending' but should be 'Running'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/chaoslib/activity.py", line 224, in run_activity
    result = run_python_activity(activity, configuration, secrets)
  File "/usr/lib/python3.7/site-packages/chaoslib/provider/python.py", line 62, in run_python_activity
    sys.exc_info()[2])
  File "/usr/lib/python3.7/site-packages/chaoslib/provider/python.py", line 57, in run_python_activity
    return func(**arguments)
  File "/usr/lib/python3.7/site-packages/chaosk8s/pod/probes.py", line 105, in pods_in_phase
    name=label_selector, s=d.status.phase, p=phase))
chaoslib.exceptions.ActivityFailed: chaoslib.exceptions.ActivityFailed: pod 'app=janie-nginx' is in phase 'Pending' but should be 'Running'

�[31m[2019-02-01 14:31:47 ERROR]�[39m => failed: chaoslib.exceptions.ActivityFailed: pod 'app=janie-nginx' is in phase 'Pending' but should be 'Running'
�[33m[2019-02-01 14:31:47 WARNING]�[39m Probe terminated unexpectedly, so its tolerance could not be validated
[2019-02-01 14:31:47 CRITICAL] Steady state probe 'pods_in_phase' is not in the given tolerance so failing this experiment
�[32m[2019-02-01 14:31:47 INFO]�[39m Let's rollback...
�[32m[2019-02-01 14:31:47 INFO]�[39m No declared rollbacks, let's move on.
�[32m[2019-02-01 14:31:47 INFO]�[39m Experiment ended with status: deviated
�[32m[2019-02-01 14:31:47 INFO]�[39m The steady-state has deviated, a weakness may have been discovered
�[36m[2019-02-01 14:31:47 DEBUG]�[39m Cleaning up controls
�[36m[2019-02-01 14:31:47 DEBUG]�[39m Clearing activities cache

Certificate authority for Kube API client is set wrongly

chaosk8s/__init__.py line 91 has:

configuration.cert_file = lookup("KUBERNETES_CA_CERT_FILE")

and
chaosk8s/__init__.py line 100 - 101 have:

configuration.cert_file = lookup("KUBERNETES_CERT_FILE")
configuration.key_file = lookup("KUBERNETES_KEY_FILE")

I believe line 91 is a bug since Kube client configuration takes ssl_ca_cert for CA cert.

PR #93 should fix it.

Could maintainers take a look at it, please?

Thanks!

Try to support old deployment api endpoint

Can we be backward compatible with very old cluster that don't support apps/v1 endpoint for deployment?

see #65

Experiment does not fails/breaks when k8s action cannot be performed

I tried to use the terminate_pods action, but without enough permissions on my K8s cluster.
However, the experiment did go till the end with "completed" status.

Is it intended behavior when a function breaks that the CTK continues other methods ?

I would have thought to have a failure at some point

[2020-10-01 12:54:06 INFO] Action: terminate-pods
[2020-10-01 12:54:06 ERROR]   => failed: kubernetes.client.rest.ApiException: (403)
    Reason: Forbidden
    HTTP response headers: HTTPHeaderDict({'Audit-Id': '1c05c067-58fc-4d95-ae0b-240ed4c3709d', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Thu, 01 Oct 2020 12:54:06 GMT', 'Content-Length': '310'})
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:chaostoolkit-run:chaostoolkit-bsm9b\" cannot list resource \"pods\" in API group \"\" in the namespace \"chaosiq-console\"","reason":"Forbidden","details":{"kind":"pods"},"code":403}
[2020-10-01 12:54:06 INFO] Pausing after activity for 2s...
[2020-10-01 12:54:11 INFO] Steady state hypothesis: Application is normal
[2020-10-01 12:54:13 INFO] Probe: application-must-respond-normally
[2020-10-01 12:54:15 INFO] Steady state hypothesis is met!
[2020-10-01 12:54:16 INFO] Let's rollback...
[2020-10-01 12:54:18 INFO] No declared rollbacks, let's move on.
[2020-10-01 12:54:19 INFO] Experiment ended with status: completed

update api version for chaosk8s extension

Latest chaosk8s version is currently looking for apis in kube cluster with version apps/v1beta1 as per code here.
Latest version of kubectl accepts only apps/v1 version for the apis.
We should upgrade it to the latest version to be compatible with latest kubectl version.

terminate_pods with tries to match labels even when one is not set.

When using terminate_pods and defining only a NS, it will also try to match the labes to none:

Config:

method:
- name: terminate-pods
  type: action
  provider:
    func: terminate_pods
    module: chaosk8s.pod.actions
    type: python
    arguments:
      ns: 'myapp'

Error log:

[2018-10-05 03:05:15 DEBUG] Using Kubernetes context: Default
[2018-10-05 03:05:15 DEBUG] Found 0 pods labelled 'None'
[2018-10-05 03:05:15 DEBUG] Activity failed
    Traceback (most recent call last):
      File "/usr/local/lib/python3.5/dist-packages/chaoslib/provider/python.py", line 50, in run_python_activity
        return func(**arguments)
      File "/usr/local/lib/python3.5/dist-packages/chaosk8s/pod/actions.py", line 58, in terminate_pods
        pods = [pods[0]]
    IndexError: list index out of range

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/local/lib/python3.5/dist-packages/chaoslib/activity.py", line 218, in run_activity
        result = run_python_activity(activity, configuration, secrets)
      File "/usr/local/lib/python3.5/dist-packages/chaoslib/provider/python.py", line 55, in run_python_activity
        sys.exc_info()[2])
      File "/usr/local/lib/python3.5/dist-packages/chaoslib/provider/python.py", line 50, in run_python_activity
        return func(**arguments)
      File "/usr/local/lib/python3.5/dist-packages/chaosk8s/pod/actions.py", line 58, in terminate_pods
        pods = [pods[0]]
    chaoslib.exceptions.ActivityFailed: IndexError: list index out of range
[2018-10-05 03:05:15 ERROR]   => failed: IndexError: list index out of range

Setting a label_selector however, gets things to work.

Allow label selector to be an argument

Currently, the way resources like pods and services are selected is via an hardcoded label_selector. It is too restrictive in most cases. Make the label_selector an argument for the activities.

Kubernetes exception

Hi,

I use my ~/.kube config to run my chaos test, this is my probe:

            {
                name: am-is-healty,
                type: probe,
                "tolerance": "0",
                provider: {
                    type: python,
                    module: chaosk8s.probes,
                    func: microservice_available_and_healthy,
                    secrets: [kubernetes],
                    arguments: {
                        name: am-6c7b66b4c8-4wmph,
                        ns: default
                    }
                }
            }

And this is the error which I'm unable to figure out:

[2019-03-25 13:59:34 DEBUG] [__init__:69] Using Kubernetes context: default
2019-03-25 13:59:50,717 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f66e109ecf8>: Failed to establish a new connection: [Errno -2] Name or service not known',)': /apis/apps/v1beta1/namespaces/default/deployments?labelSelector=name+in+%28am-6c7b66b4c8-4wmph%29

Question: when doing chao init, it stuck at defining an experimental method

I use chaostoolkit-kubernetes
when doing chao init, it stuck at defining an experimental method like this:

what should I press to choose this? I see docs about chaos init, but that is not the same as above.

How to validate internal services?

Hello Team,
I have 2 services running inside K8s but they are not exposed to the outer world through ingress/loadbalancer. We are using it inside K8s with gRPC. Is there a way to test the service endpoints for those services or "microservice_available_and_healthy" is enough?

Is there a way to check tolerance against read_pod_logs?

Hello,
I wanted to check logs returned by read_pod_logs with regex but getting the error: "return rx.search(value) is not None".
My probe was similar to:

{
   "name": "read-microservices-logs",
   "type": "probe",
   "tolerance": {
      "type": "regex,",
      "pattern": "UP"
   },
   "provider": {
      "func": "read_pod_logs",
      "arguments": {
         "label_selector": "app.kubernetes.io/name=ui",
         "ns": "dev",
         "container_name": "ui"
      },
      "module": "chaosk8s.probes",
      "type": "python"
   }
}

But got an error:
File "/opt/hostedtoolcache/Python/3.8.1/x64/lib/python3.8/site-packages/chaoslib/hypothesis.py", line 304, in _ return rx.search(value) is not None TypeError: expected string or bytes-like object

Can it be caused by function returning dictionary instead of string?

SSLCertVerificationError while trying to run experiment againts a remote k8s cluster

Steps:

1. Installed chaostoolkit-kubernetes using pip install chaostoolkit-kubernetes command on my local windows machine
2. Have a remote kubernetes cluster running. Added the kube config file pointing to the remote k8s cluster in C:\Users<xxx>.kube folder on local machine
3. Verified kube config file by making running "kubectl" commands.
4. Created a simple experiment as shown https://docs.chaostoolkit.org/drivers/kubernetes/#usage
5. Execute chaos run by passing the experiment file.

Output:
[2020-09-01 18:52:53 INFO] Validating the experiment's syntax
[2020-09-01 18:52:53 INFO] Experiment looks valid
[2020-09-01 18:52:53 INFO] Running experiment: Do we remain available in face of pod going down?
[2020-09-01 18:52:53 INFO] Steady state hypothesis: Verifying service remains healthy
[2020-09-01 18:52:53 INFO] Probe: all-our-microservices-should-be-healthy
[2020-09-01 18:52:57 ERROR] => failed: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='xxxxxxxx', port=443): Max retries exceeded with url: /k8s/clusters/xxxxx/apis/apps/v1/namespaces/xxxxx/deployments?fieldSelector=metadata.name%3Dxxxxx (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)')))
[2020-09-01 18:52:57 WARNING] Probe terminated unexpectedly, so its tolerance could not be validated
[2020-09-01 18:52:57 CRITICAL] Steady state probe 'all-our-microservices-should-be-healthy' is not in the given tolerance so failing this experiment
[2020-09-01 18:52:57 INFO] Experiment ended with status: failed

Can someone help what could be missing?

Move read_microservices_logs to pod.probes

That probe is probably better located in chaosk8s.pod.probes.

Drain Node Error

Receive the following error when running drain_nodes function.

`failed: TypeError: Got an unexpected keyword argument 'include_uninitialized' to method list_pod_for_all_namespaces'

Installed chaostoolkit-kubernetes using pip3 with Python 3.7.4. Using version chaostoolkit version 1.2.0

Below is the action I'm using:

  39   │     "method": [
  40   │         {
  41   │             "type": "action",
  42   │             "name": "drain_node",
  43   │             "provider": {
  44   │                 "type": "python",
  45   │                 "module": "chaosk8s.node.actions",
  46   │                 "func": "drain_nodes",
  47   │                 "arguments": {
  48 ~ │                     "name": "aks-agentpool-24917760-1"
  49   │                 }
  50   │             }
  51   │         }
  52   │     ],

Ability to specify a maximum percentage of pods to be killed

Currently, chaostoolkit-kubernetes has a method terminate_pods that supports to kill only one pod or all pods at a time (with and without randomization). It will be great if there is a provision to specify a percentage of pods that can be killed.

kill_microservice not working correctly

similar to #42

Code:

"type": "action",
            "name": "stop-provider-service",
            "provider": {
                "type": "python",
                "module": "chaosk8s.actions",
                "func": "kill_microservice",
                "arguments": {
                    "name": "my-provider-app"
                }

Output:

[2019-08-30 09:40:44 ERROR]   => failed: TypeError: delete_namespaced_deployment() takes 3 positional arguments but 4 were given

I got this code from the Sample github but the kill_microservice function does not seem to be working correctly.

Allow users to specify context for config

Allow passing the Kubernetes context so that we use the right one when connecting.

As per aws-samples/aws-workshop-for-kubernetes#428

AttributeError on pytest

OS: Win10
Python: 3.7.5 But it's the same for 3.7.7

I'm facing a problem when I try to execute pytest

λ pytest
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "c:\users\devil\appdata\local\programs\python\python37\lib\site-packages\_pytest\main.py", line 187, in wrap_session
INTERNALERROR>     config._do_configure()
INTERNALERROR>   File "c:\users\devil\appdata\local\programs\python\python37\lib\site-packages\_pytest\config\__init__.py", line 820, in _do_configure
INTERNALERROR>     self.hook.pytest_configure.call_historic(kwargs=dict(config=self))
INTERNALERROR>   File "c:\users\devil\appdata\local\programs\python\python37\lib\site-packages\pluggy\hooks.py", line 308, in call_historic
INTERNALERROR>     res = self._hookexec(self, self.get_hookimpls(), kwargs)
INTERNALERROR>   File "c:\users\devil\appdata\local\programs\python\python37\lib\site-packages\pluggy\manager.py", line 93, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "c:\users\devil\appdata\local\programs\python\python37\lib\site-packages\pluggy\manager.py", line 87, in <lambda>
INTERNALERROR>     firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
INTERNALERROR>   File "c:\users\devil\appdata\local\programs\python\python37\lib\site-packages\pluggy\callers.py", line 208, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "c:\users\devil\appdata\local\programs\python\python37\lib\site-packages\pluggy\callers.py", line 80, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "c:\users\devil\appdata\local\programs\python\python37\lib\site-packages\pluggy\callers.py", line 187, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "C:\Users\devil\AppData\Local\Programs\Python\Python37\lib\site-packages\pytest_sugar.py", line 176, in pytest_configure
INTERNALERROR>     sugar_reporter = SugarTerminalReporter(standard_reporter)
INTERNALERROR>   File "C:\Users\devil\AppData\Local\Programs\Python\Python37\lib\site-packages\pytest_sugar.py", line 214, in __init__
INTERNALERROR>     self.writer = self._tw
INTERNALERROR> AttributeError: can't set attribute

λ python --version
Python 3.7.5

λ pip --version
pip 20.0.2 from c:\users\devil\appdata\local\programs\python\python37\lib\site-packages\pip (python 3.7)

It's happening also on travis : https://github.com/tdevilleduc/chaostoolkit-kubernetes/runs/592029162

I've manage to get rid of this early today but reinstall python (downgrade to 3.7.5) but it appends again. The only thing I've done by then is to rebase my repo from this one

Did you ever add this behaviour ?

Cannot use no_proxy env variable with http_proxy

I work behind a corporate proxy with windows 10
So i use 3 env variables:

• HTTP_PROXY
• HTTPS_PROXY
• NO_PROXY in order to specify some ip and host that need to bypass proxy definition

When i'm using chaostoolkit-kubernetes i cannot contact my minikube cluster

[2019-12-20 15:55:19 INFO] Validating the experiment's syntax
[2019-12-20 15:55:20 INFO] Experiment looks valid
[2019-12-20 15:55:20 INFO] Running experiment: Le pod MySQL KO ne doit pas perturber le service
[2019-12-20 15:55:20 INFO] Steady state hypothesis: Les APIs répondent
[2019-12-20 15:55:20 INFO] Probe: Swagger OK
[2019-12-20 15:55:20 INFO] Probe: GET /api/users OK
[2019-12-20 15:55:20 INFO] Probe: GET /api/users/{id} OK
[2019-12-20 15:55:20 INFO] Probe: GET /api/workstations OK
[2019-12-20 15:55:20 INFO] Probe: GET /api/workstations/{id} OK
[2019-12-20 15:55:20 INFO] Steady state hypothesis is met!
[2019-12-20 15:55:20 INFO] Action: Suppression du pod MYSQL
2019-12-20 15:55:23,855 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden'))': /apis/apps/v1/namespaces/default/deployments?labelSelector=module%3Dmysql
2019-12-20 15:55:23,911 WARNING Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden'))': /apis/apps/v1/namespaces/default/deployments?labelSelector=module%3Dmysql
2019-12-20 15:55:23,998 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden'))': /apis/apps/v1/namespaces/default/deployments?labelSelector=module%3Dmysql
[2019-12-20 15:55:24 ERROR] => failed: urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='145.42.9.20', port=8443): Max retries exceeded with url: /apis/apps/v1/namespaces/default/deployments?labelSelector=module%3Dmysql (Caused by ProxyError('Cannot connect to proxy.', OSError('Tunnel connection failed: 403 Forbidden')))
[2019-12-20 15:55:24 INFO] Steady state hypothesis: Les APIs répondent
[2019-12-20 15:55:24 INFO] Probe: Swagger OK
[2019-12-20 15:55:24 INFO] Probe: GET /api/users OK
[2019-12-20 15:55:24 INFO] Probe: GET /api/users/{id} OK
[2019-12-20 15:55:24 INFO] Probe: GET /api/workstations OK
[2019-12-20 15:55:24 INFO] Probe: GET /api/workstations/{id} OK
[2019-12-20 15:55:24 INFO] Steady state hypothesis is met!
[2019-12-20 15:55:24 INFO] Let's rollback...
[2019-12-20 15:55:24 INFO] No declared rollbacks, let's move on.
[2019-12-20 15:55:24 INFO] Experiment ended with status: completed

The only way to make this work is too remove HTTP_PROXY env variable
But I need it for other use cases

Is it possible to have NO_PROXY en variable support inchaostoolkit-kubernetes ?
I tried to make it myself but i'm newbie in Python

404 error on chaosk8s action [kill, remove, scale]

I am seeing issues when I am trying to run experiment on micro service hosted on minikube .
I see this behavior for actions which accepts name as mandatory parameter in the request:

Here is the link to my GitHub repository which host the code for sample spring boot micro service that I am deploying on local minikube and trying to run chaos experiments. All the probes are successful . Only experiments are failing.

Error when running chaos:

kubect service description:

Error on executing chaos commands in existing K8 cluster

tejendrabhandari@cloudshell:/chaostoolkitexp/go-demo-8 (fluted-polymer-273010)$ kubectl apply -f k8s/terminate-pods/pod.yaml -n go-demo-8
pod/go-demo-8 created
tejendrabhandari@cloudshell:/chaostoolkitexp/go-demo-8 (fluted-polymer-273010)$ pip install -U chaostoolkit-kubernetes

Python 2 is deprecated. Upgrade to pip3 as soon as possible.
See https://cloud.google.com/python/docs/python2-sunset

To suppress this warning, create an empty ~/.cloudshell/no-pip-warning file.
The command will automatically proceed in 5 seconds or on any key.

DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Defaulting to user installation because normal site-packages is not writeable
Collecting chaostoolkit-kubernetes
Downloading chaostoolkit_kubernetes-0.1.0-py2.py3-none-any.whl (5.6 kB)
Requirement already satisfied, skipping upgrade: pycodestyle in /usr/local/lib/python2.7/dist-packages (from chaostoolkit-kubernetes) (2.5.0)
Collecting chaostoolkit-lib
Downloading chaostoolkit_lib-0.1.0-py2.py3-none-any.whl (15 kB)
Requirement already satisfied, skipping upgrade: pyyaml in /home/tejendrabhandari/.local/lib/python2.7/site-packages (from chaostoolkit-kubernetes) (5.3.1)
Requirement already satisfied, skipping upgrade: kubernetes in /home/tejendrabhandari/.local/lib/python2.7/site-packages (from chaostoolkit-kubernetes) (11.0.0)
Requirement already satisfied, skipping upgrade: logzero in /home/tejendrabhandari/.local/lib/python2.7/site-packages (from chaostoolkit-lib->chaostoolkit-kubernetes) (1.5.0)....
....
Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /usr/local/lib/python2.7/dist-packages (from requests->kubernetes->chaostoolkit-kubernetes) (2.9)
Requirement already satisfied, skipping upgrade: pyasn1>=0.1.3 in /usr/local/lib/python2.7/dist-packages (from rsa<4.1,>=3.1.4->google-auth>=1.0.1->kubernetes->chaostoo
lkit-kubernetes) (0.4.8)
Installing collected packages: chaostoolkit-lib, chaostoolkit-kubernetes
Successfully installed chaostoolkit-kubernetes-0.1.0 chaostoolkit-lib-0.1.0

after succssfull install
Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in /usr/local/lib/python2.7/dist-packages (from requests->kubernetes->chaostoolkit-kubernetes) (2.9)
Requirement already satisfied, skipping upgrade: pyasn1>=0.1.3 in /usr/local/lib/python2.7/dist-packages (from rsa<4.1,>=3.1.4->google-auth>=1.0.1->kubernetes->chaostoolkit-kubernetes) (0.4.8)
Installing collected packages: chaostoolkit-lib, chaostoolkit-kubernetes
Successfully installed chaostoolkit-kubernetes-0.1.0 chaostoolkit-lib-0.1.0