powerfulseal / powerfulseal Goto Github PK

A powerful testing tool for Kubernetes clusters.

License: Apache License 2.0

Makefile 0.54% Python 98.39% Dockerfile 0.50% Shell 0.03% JavaScript 0.54%

kubernetes testing openstack chaos-engineering hacktoberfest

powerfulseal's Introduction

PowerfulSeal

PowerfulSeal injects failure into your Kubernetes clusters, so that you can detect problems as early as possible. It allows for writing scenarios describing complete chaos experiments.

Embrace the inevitable failure. Embrace The Seal.

Documentation

Please refer to the Powerfulseal documentation to learn how to use it.

Highlights

works with Kubernetes, OpenStack, AWS, Azure, GCP and local machines
yaml policies describing complete chaos experiments
Prometheus and Datadog metrics collection
multiple modes for differnt use cases

Hello world!

Just to give you a taste, here's an example policy. It will kill a single pod, and then check that the service continues responding to HTTP probes, to verify its resiliency to one of its pods going down.

scenarios:
- name: Kill one pod in my namespace, make sure the service responds
  steps:
  # kill a pod from `myapp` namespace
  - podAction:
      matches:
        - namespace: myapp
      filters:
        - randomSample:
            size: 1
      actions:
        - kill:
            probability: 0.75
  # check my service continues working
  - probeHTTP:
      target:
        service:
          name: my-service
          namespace: myapp
      endpoint: /healthz

Assuming that's in policy.yml, you can run it like this:

powerfulseal autonomous --policy-file ./policy.yaml

Learn more

Installing

docker hub: docker pull powerfulseal/powerfulseal:3.1.1
pip: pip install powerfulseal

Read about the PowerfulSeal

Tools consuming PowerfulSeal

Chaos and resiliency testing tool for Kubernetes and OpenShift: https://github.com/openshift-scale/kraken

Footnotes

powerfulseal's People

Stargazers

Watchers

Forkers

kconst fjibj alanfx sidcarter aelmadho waleadedeji cclauss sejeff tikyau prateeknayak ekuric alexandrst88 dogopupper alexxnica kryndex dc-project duke-lv lmeyemezu torubylist suntofly onlymellb alezkv ngaut hhxk18 anchorfree etsangsplk stakater joe9724 mkartashov kajanthm joshuakwan chenqiangzhishen kz caniszczyk sgnanask ben-haim sbambach scostache kunalpatelibm stefanhipfel gronau-it-cloud-computing ljakimczuk nurturingmyself macbash rgodishela nonspecialist ben-paton hunglethanh9 pkpattabhiraman snehalbiche qiell sesteva stelligent onophris zhanghuidinah syd-vigilant curx robinsea dimpu47 frantheitalian xiaoruiguo april4 mikeln sudahang twexler stretchcloud dgzlopes ruteri skyionblue matt-mahdieh patricioreyna ksatchit xorilog brentley stephanzf gaborbernat raghu999 mhamdisemah fahedouch seeker89 saokar arseniipetrovich sakuralbj muvvas darkrat lichnak pathcl pushkar-bitwise andypeng2015 jpkubernetes avik1993 jpenekusu adamnugraha hassenius jeblackmore jianjian12138 mattantley benluteijn sunilpentapati aquibchiniwala

powerfulseal's Issues

Setup Travis CI

Fail to execute docker command

root@node1:~#seal --kube-config /root/kubeconfig --interactive --inventory-kubernetes --no-cloud

(seal) $ docker
Executing '['sh', '-c', 'sudo docker ']' on local-172.24.26.66
--------------------------------------------------------------------------------
172.24.26.66


Error creating SSH connection
Original error: Authentication failed.
Executing '['sh', '-c', 'sudo docker ']' on local-172.24.26.67
--------------------------------------------------------------------------------
172.24.26.67


Error creating SSH connection
Original error: Authentication failed.
Executing '['sh', '-c', 'sudo docker ']' on local-172.24.26.68
--------------------------------------------------------------------------------
172.24.26.68


Error creating SSH connection
Original error: Authentication failed.

Kill command returned 'Node not found'

Hello, I installed powerfulseal by pip in Ubuntu VM that was installed k8s.I ran into interactive mode, but It returns nothing when I executed kill pod, Is there any wrong or missed step?

root@taco-aio:~# powerfulseal --interactive --inventory-kubernetes --kube-config .kube/config
(seal) $ groups
amd64
enabled
linux
taco-aio
true
(seal) $ namespaces
ceph
default
kube-public
kube-system
openstack
(seal) $ pods openstack application=nova
[pod #0 name=nova-api-metadata-58d96477bc-5m8pf namespace=openstack containers=1 ip=172.16.106.22 host_ip=192.168.122.190 state
=Running labels:application=nova,pod-template-hash=1485203367,component=metadata,release_group=taco-nova]
[pod #1 name=nova-api-osapi-658fdf8688-knx9r namespace=openstack containers=1 ip=172.16.106.24 host_ip=192.168.122.190 state=Ru
nning labels:application=nova,pod-template-hash=2149894244,component=os-api,release_group=taco-nova]
[pod #2 name=nova-bootstrap-p6gw8 namespace=openstack containers=1 ip=172.16.106.49 host_ip=192.168.122.190 state=Succeeded lab
els:application=nova,controller-uid=cb019bea-0bbf-11e8-bca1-525400d69f89,component=bootstrap,job-name=nova-bootstrap,release_gr
oup=taco-nova]
[pod #3 name=nova-cell-setup-nmjk8 namespace=openstack containers=1 ip=172.16.106.28 host_ip=192.168.122.190 state=Succeeded la
bels:application=nova,controller-uid=cb039bf3-0bbf-11e8-bca1-525400d69f89,component=cell-setup,job-name=nova-cell-setup,release
_group=taco-nova]
[pod #4 name=nova-compute-5b9sx namespace=openstack containers=2 ip=192.168.122.190 host_ip=192.168.122.190 state=Running label
s:application=nova,pod-template-generation=1,component=compute,controller-revision-hash=3918579215,release_group=taco-nova]
[pod #5 name=nova-conductor-6cd6689fb9-6554f namespace=openstack containers=1 ip=172.16.106.44 host_ip=192.168.122.190 state=Ru
nning labels:application=nova,pod-template-hash=2782245965,component=conductor,release_group=taco-nova]
[pod #6 name=nova-consoleauth-f44969c95-tc9mx namespace=openstack containers=1 ip=172.16.106.35 host_ip=192.168.122.190 state=R
unning labels:application=nova,pod-template-hash=900525751,component=consoleauth,release_group=taco-nova]
[pod #7 name=nova-db-init-vbwrw namespace=openstack containers=3 ip=172.16.106.3 host_ip=192.168.122.190 state=Succeeded labels
:application=nova,controller-uid=cb07cca8-0bbf-11e8-bca1-525400d69f89,component=db-init,job-name=nova-db-init,release_group=tac
o-nova]
[pod #8 name=nova-db-sync-llvqc namespace=openstack containers=1 ip=172.16.106.41 host_ip=192.168.122.190 state=Succeeded label
s:application=nova,controller-uid=cb0c9fe1-0bbf-11e8-bca1-525400d69f89,component=db-sync,job-name=nova-db-sync,release_group=ta
co-nova]
[pod #9 name=nova-ks-endpoints-29zkm namespace=openstack containers=3 ip=172.16.106.7 host_ip=192.168.122.190 state=Succeeded l
abels:application=nova,controller-uid=cb11dcb7-0bbf-11e8-bca1-525400d69f89,component=ks-endpoints,job-name=nova-ks-endpoints,re
lease_group=taco-nova]
[pod #10 name=nova-ks-service-n2kcn namespace=openstack containers=1 ip=172.16.106.2 host_ip=192.168.122.190 state=Succeeded la
bels:application=nova,controller-uid=cb3709d5-0bbf-11e8-bca1-525400d69f89,component=ks-service,job-name=nova-ks-service,release
_group=taco-nova]
[pod #11 name=nova-ks-user-9lj5m namespace=openstack containers=1 ip=172.16.106.51 host_ip=192.168.122.190 state=Succeeded labe
ls:application=nova,controller-uid=cb3b1e7c-0bbf-11e8-bca1-525400d69f89,component=ks-user,job-name=nova-ks-user,release_group=t
aco-nova]
[pod #12 name=nova-novncproxy-7db5fb96b9-xtwph namespace=openstack containers=1 ip=192.168.122.190 host_ip=192.168.122.190 stat
e=Running labels:application=nova,pod-template-hash=3861965265,component=novnc-proxy,release_group=taco-nova]
[pod #13 name=nova-placement-api-5b794b96f8-dfhr2 namespace=openstack containers=1 ip=172.16.106.10 host_ip=192.168.122.190 sta
te=Running labels:application=nova,pod-template-hash=1635065294,component=placement,release_group=taco-nova]
[pod #14 name=nova-scheduler-59b5999769-jnr2n namespace=openstack containers=1 ip=172.16.106.9 host_ip=192.168.122.190 state=Ru
nning labels:application=nova,pod-template-hash=1561555325,component=scheduler,release_group=taco-nova]
(seal) $ kill 14
Node not found
(seal) $

Error: Openshift

I'm getting an error message when running this command bellow: I'm using Openshift which shouldn't really matter.

powerfulseal --run-policy-file example.policy --kube-config ~/.kube/config --inventory-kubernetes --no-cloud -v

ERROR:powerfulseal.clouddrivers.no_cloud_driver:Trying to sync things while using a no-cloud driver. If you don't expect to be seeing this, you might want to rethink some of your choices
Executing '['sh', '-c', 'sudo docker kill -s SIGKILL bb549a611670cc65322bacb4b674c11f0dd9e5c69732e13c93999d4860093206']' on local-10.0.2.15
Pod killed - namespace: metrics - name: alertmanager-metrics-0
Executing '['sh', '-c', 'sudo docker kill -s SIGKILL 6dd2bb3613fbf805ac3ae1e0775c341b500c7458f88319282ae7cced1cd06875']' on local-10.0.2.15
Pod killed - namespace: metrics - name: kube-state-metrics-6b67c7b958-6ccr2
Executing '['sh', '-c', 'sudo docker kill -s SIGKILL cb569d3e3f12c80fc7ad8d25ac97b80244a893d9dd029b7be73eadfb4d778233']' on local-10.0.2.15
Pod killed - namespace: metrics - name: node-exporter-stg5k
Executing '['sh', '-c', 'sudo docker kill -s SIGKILL 39798cab85e96f30af3f4c79bf83bb2c7591650b8e5eba354feb4f984eec704a']' on local-10.0.2.15
Pod killed - namespace: metrics - name: kubernetes-pod-chaos-monkey-b9879fb44-jw4mg
Executing '['sh', '-c', 'sudo docker kill -s SIGKILL 2ad332b2cae785608ed24d8a0bbc5b34240366b60091126ce2e0600bf1dc0564']' on local-10.0.2.15

My Policy is very simple:

podScenarios:
 - name: "delete random pods"

   match:
     - namespace:
         name: "metrics"
   filters:
     - property:
         name: "state"
         value: "Running"
   actions:
     - kill:
         probability: 1
         force: true
     - wait:
         seconds: 5

Additional steps for Oracle Linux 7.3, Python 2.7.5 for installing dependencies and compilation issues

On my Oracle Linux 7.3 box with Python 2.7.5, I had to perform some additional steps to get powerfulseal to work.
$ sudo pip install powerfulseal
Installing collected packages: ConfigArgParse, click, MarkupSafe, Jinja2, Werkzeug, itsdangerous, Flask, termcolor, pyparsing, packaging, deprecation, iso8601, pbr, stevedore, keystoneauth1, os-service-types, munch, netifaces, futures, appdirs, requestsexceptions, jmespath, jsonpointer, jsonpatch, dogpile.cache, openstacksdk, bcrypt, asn1crypto, cryptography, pynacl, paramiko, spur, python-dateutil, pyasn1-modules, rsa, httplib2, oauth2client, kubernetes, powerfulseal
Running setup.py install for ConfigArgParse ... done
Running setup.py install for MarkupSafe ... done
Running setup.py install for itsdangerous ... done
Running setup.py install for termcolor ... done
Found existing installation: pyparsing 1.5.6 DEPRECATION: Uninstalling a distutils installed project (pyparsing) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
Uninstalling pyparsing-1.5.6: Successfully uninstalled pyparsing-1.5.6
Running setup.py install for deprecation ... done
Running setup.py install for munch ... done
Running setup.py install for netifaces ... done
Running setup.py install for dogpile.cache ... done
Found existing installation: cryptography 1.3.1 Uninstalling cryptography-1.3.1: Successfully uninstalled cryptography-1.3.1
Running setup.py install for httplib2 ... done
Running setup.py install for powerfulseal ... done
Successfully installed ConfigArgParse-0.13.0 Flask-0.12.2 Jinja2-2.10 MarkupSafe-1.0 Werkzeug-0.14.1 appdirs-1.4.3 asn1crypto-0.24.0 bcrypt-3.1.4 click-6.7 cryptography-2.2.2 deprecation-2.0.2 dogpile.cache-0.6.5 futures-3.2.0 httplib2-0.11.3 iso8601-0.1.12 itsdangerous-0.24 jmespath-0.9.3 jsonpatch-1.23 jsonpointer-2.0 keystoneauth1-3.5.0 kubernetes-1.0.2 munch-2.3.1 netifaces-0.10.6 oauth2client-4.1.2 openstacksdk-0.12.0 os-service-types-1.2.0 packaging-17.1 paramiko-2.4.1 pbr-4.0.2 powerfulseal-1.1.1 pyasn1-modules-0.2.1 pynacl-1.2.1 pyparsing-2.2.0 python-dateutil-2.7.2 requestsexceptions-1.4.0 rsa-3.4.2 spur-0.3.20 stevedore-1.28.0 termcolor-1.1.0

after I completed the installation for powerfulseal (as per logs above), I wasn't able to execute powerfulseal.
In order to resolve the issues, these are the steps that I had to perform -

sudo pip install pyasn1 --upgrade
sudo pip install cffi --upgrade
sudo pip install configparser
File: "/usr/lib/python2.7/site-packages/powerfulseal/cli/pscmd.py" - Added the line below to the top of the file.
from __future__ import print_function
File: /usr/lib/python2.7/site-packages/powerfulseal/clouddrivers/driver.py - Added the first two lines to import six for compatibility and updated the code as shown in line 3 below.
import abc, six
@six.add_metaclass(abc.ABCMeta)
class AbstractDriver():

Add tests to command line interpreter to prevent regressions

Expose a maximum number of loops for autonomous mode

The PolicyRunner.run method supports a maximum number of times to run (loops= named parameter) but this is not exposed as far as I can see in a useful manner.

A CLI flag of the maximum number of loops would be useful for testers/scripters who'd like to run PS scenarios a set number of times and then exit, to present test run results.

(I have this exposure implemented privately, and am willing to contribute it, but wanted to get the idea vetted first)

Add Azure Cloud Driver

Add an Azure Cloud Driver.

Kubernetes-only mode

Currently, to run the PowerfulSeal, it is necessary to provide cloud provider details.

The seal could be more useful if it had an extra mode for constricted environments, in which it only speaks to Kubernetes, and/or Kubernetes + SSH. This way, it could provide more value in the corner cases.

Use the logging module instead of print() throughout the codebase

Because it is more better-er.

Filter based on allowed match mappings

Scenario: multiple pod-specific matchers are used to match pods (namespace and deployment) and I wish to use a filter of the same criteria to effectively make the match an AND operation instead of OR

Currently there is no way to filter on these pod-specific matchers, being instead limited to property/dayTime/randomSample/probability (populated in the filter mapping in scenario.py)

I'd like to propose adding the capability for pod_scenario and node_scenario to optionally add to the filter mapping to provide their own filters for cases that make sense.

Negative match and filter logic

Scenario: powerfulseal is hosted in a cluster it is chaosing. It would be useful to be able negative match and filter logic - that is the ability to say "match nodes that are NOT ip-foo-bar" and "filter for nodes that are NOT ip-foo-bar" to allow powerfulseal to do safer node kills in this situation

Because property_match uses a regex SOME of this functionality could, with some work, be done. The addition of a property_not mapping, however, would provide easier negation logic for policy writers

On-premise ipmi "cloud provider"

How would you envision using powerfulseal for a bare metal on premise kubernetes environment? All of the servers have IPMI access for remote power. Would it be a new cloud driver, or would it be an addition to #17 along with some ssh magic? I might be able to work on some of it with an idea of how this should be implemented with the idea of getting it back upstream.

Wrong import breaking cli

Hi!

I wanted to try powerfulseal, but I found that after pip-installing it I cannot even run the help command. It seems there's a dependency issue in the python package (you may want to check the ci job of master, it should be failing).

I noticed os_client_config is now embedded into openstack package, so a simple patch like the following would do.


diff --git a/powerfulseal/clouddrivers/open_stack_driver.py b/powerfulseal/clouddrivers/open_stack_driver.py
index 71708a1..c343f08 100644
--- a/powerfulseal/clouddrivers/open_stack_driver.py
+++ b/powerfulseal/clouddrivers/open_stack_driver.py
@@ -15,15 +15,14 @@


 import logging
-import os_client_config
-from openstack import connection
 from . import AbstractDriver
 from ..node import Node, NodeState


 def create_connection_from_config(name=None):
     """ Creates a new open stack connection """
-    occ = os_client_config.OpenStackConfig()
+    from openstack import connection, OpenStackConfig
+    occ = OpenStackConfig()
     cloud = occ.get_one_cloud(name)
     return connection.from_config(cloud_config=cloud)

@@ -106,4 +105,3 @@ class OpenStackDriver(AbstractDriver):
         """ Delete a Node permanently.
         """
         self.conn.compute.delete_server(node.id)
-

if the cloud provider is not aws,such aliyun,gcloud,azure, should user need to develop driver for other cloud?

Running with `--inventory-kubernetes` fails to find host in known_hosts

Summary

In interactive mode, for certain commands, the powerful seal tries to ssh into a destination host to run the command. Using the --inventory-kubernetes flag makes the seal try to guess the inventory from kubernetes but if a host has both a public IP address and a private IP address, it tends to find the public IP address and then have ssh complain about the IP address not being present in known_hosts.

Steps to reproduce

Run powerfulseal with powerfulseal --inventory-kubernetes --interactive --kube-config /kubernetes/conf/kubeconfig
Give the command docker. We get an error of the form:

Executing '['sh', '-c', 'sudo docker ']' on sk1-worker-1
--------------------------------------------------------------------------------
xxx.xxx.xxx.xxx


Error creating SSH connection
Original error: Server 'xxx.xxx.xxx.xxx' not found in known_hosts

Where xxx.xxx.xxx.xxx is the public IP address of a host that is present in known hosts.

Can I use powerfulseal in a private cloud?

Whether to support？

Validating a policy file should not require configuring the cloud or kubernetes drivers

For instance, this should be allowed to be executed:

$ seal --validate-policy-file ./example_config.yml
usage: seal [-h] [-c CONFIG] [-v] (-i INVENTORY_FILE | --inventory-kubernetes)
            [--open-stack-cloud OPEN_STACK_CLOUD] [--remote-user REMOTE_USER]
            [--kube-config KUBE_CONFIG]
            (--validate-policy-file VALIDATE_POLICY_FILE | --run-policy-file RUN_POLICY_FILE | --interactive)
seal: error: one of the arguments -i/--inventory-file --inventory-kubernetes is required

policy file configuration

Hi,
need more explanation on how this works in total?
my expectation was after setting below in policy file powerfulseal will start killing my pods once the start time hits and will run for 24 hours. but that never the case it start with saying too early and then continuously shows too late, why?

current time was :- 12: 01
The day of week :- Monday

My current configuration in policy file as below -
- dayTime:
onlyDays:
- "monday"
- "tuesday"

- "wednesday"

- "thursday"

- "friday"

- "saturday"

- "sunday"

     startTime:
        hour: 12
        minute: 03
        second: 00
      endTime:
        hour: 11
        minute: 50
        second: 00

From logs -

INFO:powerfulseal.cli.main:Attempting to read the inventory from kubernetes
ERROR:powerfulseal.clouddrivers.no_cloud_driver:Trying to sync things while using a no-cloud driver. If you don't expect to be seeing this, you might want to rethink some of your choices
INFO:powerfulseal.policy.scenario.The Chaos Testing:Matched 1 pods in namespace powerfulsealtest
INFO:powerfulseal.policy.scenario.The Chaos Testing:Matching [pod #0 name=sise-748bf8d98b-w29qk namespace=powerfulsealtest containers=1 ip= host_ip= state=Running labels:owner=luser,pod-template-hash=3046948546,run=sise]
INFO:powerfulseal.policy.scenario.The Chaos Testing:Initial set length: 1
INFO:powerfulseal.policy.scenario.The Chaos Testing:Filter property: 1 -> 1 items
INFO:powerfulseal.policy.scenario.The Chaos Testing:Filter property: 1 -> 1 items
INFO:powerfulseal.policy.scenario.The Chaos Testing:Now is datetime.datetime(2018, 9, 3, 12, 2, 9, 10619)
### INFO:powerfulseal.policy.scenario.The Chaos Testing:Too early
INFO:powerfulseal.policy.scenario.The Chaos Testing:Filter dayTime: 1 -> 0 items
INFO:powerfulseal.policy.scenario.The Chaos Testing:Empty set after {'dayTime': {'onlyDays': ['monday', 'tuesday'], 'startTime': {'second': 0, 'hour': 12, 'minute': 3}, 'endTime': {'second': 0, 'hour': 11, 'minute': 50}}}
Filtered to empty set
INFO:powerfulseal.policy.scenario.The Chaos Testing:Filtered set length: 0
INFO:powerfulseal.policy.scenario.The Chaos Testing:Done
INFO:powerfulseal.policy.policy_runner:Sleeping for 177 seconds
ERROR:powerfulseal.clouddrivers.no_cloud_driver:Trying to sync things while using a no-cloud driver. If you don't expect to be seeing this, you might want to rethink some of your choices
INFO:powerfulseal.policy.scenario.The Chaos Testing:Matched 1 pods in namespace powerfulsealtest
INFO:powerfulseal.policy.scenario.The Chaos Testing:Matching [pod #0 name=sise-748bf8d98b-w29qk namespace=powerfulsealtest containers=1 ip= host_ip= state=Running labels:owner=,pod-template-hash=3046948546,run=sise]
INFO:powerfulseal.policy.scenario.The Chaos Testing:Initial set length: 1
INFO:powerfulseal.policy.scenario.The Chaos Testing:Filter property: 1 -> 1 items
INFO:powerfulseal.policy.scenario.The Chaos Testing:Filter property: 1 -> 1 items
INFO:powerfulseal.policy.scenario.The Chaos Testing:Now is datetime.datetime(2018, 9, 3, 12, 3, 23, 774632)

INFO:powerfulseal.policy.scenario.The Chaos Testing:Too late

INFO:powerfulseal.policy.scenario.The Chaos Testing:Filter dayTime: 1 -> 0 items
INFO:powerfulseal.policy.scenario.The Chaos Testing:Empty set after {'dayTime': {'onlyDays': ['monday', 'tuesday'], 'startTime': {'second': 0, 'hour': 12, 'minute': 3}, 'endTime': {'second': 0, 'hour': 11, 'minute': 50}}}
Filtered to empty set
INFO:powerfulseal.policy.scenario.The Chaos Testing:Filtered set length: 0
INFO:powerfulseal.policy.scenario.The Chaos Testing:Done
INFO:powerfulseal.policy.policy_runner:Sleeping for 301 seconds

Enhancement: Allow Access Through Bastion Host

Add capability to ssh to the kubernetes nodes through a bastion host.

Support ssh client configs

With the current implementation of spur lib for ssh connectivity, I do not see a way for providing an ssh client config (whereby one may have a client config that uses a ProxyCommand configuration). Please consider supporting ssh client config (rather than passing a path to a private key), as modern access security models support multiple keys for access to inside hosts through bastion/edge nodes.

Will Powerful seal works for Openshift Origin ?

Hi Team,

Will Powerful seal works for the open shift origin ?

Thanks

Inconsistency naming the "percentage" parameter in configuration

The property "percentage" seems to expect a value between 0 and 1 while the name suggests a value between 0 and 100.
I think it would be clearer to use if the property was named "proportion"/"ratio" or if the scale of the value was changed.

Can we run Powerfulseal in GCP/GKE?

Hello Team,

I have a project in GCP and have Kubernetes cluster in it where in for my testing purpose I am planning to kill my pods/nodes using Powerfulseal. Can you please let me know if Powerfulseal supports with GCP/GKE.

Add an option to allow the seal to connect to a host not in your `known_hosts`

For certain commands in interactive mode, the powerful seal connects to the host via ssh. If the host is not present in your known_hosts, the seal will fail to connect and throw an error of the form:

Executing '['sh', '-c', 'sudo docker ']' on sk1-worker-1
--------------------------------------------------------------------------------
xxx.xxx.xxx.xxx


Error creating SSH connection
Original error: Server 'xxx.xxx.xxx.xxx' not found in known_hosts

There is no way to make the seal either add the IP address to your known hosts or have it ignore the known hosts file altogether. We should consider having ssh use -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no on an explicit command from the user.

Can't figure out how to provide an inventory for autonomous mode

Hi,

Can you please provide me an example of an inventory file (apart from the one in tests) that allows me to match with group properties in the node_scenarios policy. I use this now,

[c]
52.58.80.238

[a]
3.120.248.11

[workers:children]
c
a

How do I provide more info here to match to the group properties for a policy like below?

config:
  minSecondsBetweenRuns: 3600
  maxSecondsBetweenRuns: 7200

# the scenarios describing actions on nodes
nodeScenarios:

# example of a policy using al the filters available
- name: "reboot control node"

  # Choose the initial set of nodes to operate on.
  # Note that this will be an union of all the notes you match (logical OR)
  match:
  - property:
      name: "group"
      value: "a"

  # The filters are executed in the order specified and can be
  # used multiple times, and piped from one to the next.
  filters:

  # property filters (all the property filters support regexp)
  - property:
      name: "group"
      value: "a"

  # time of execution filters
  # to restrict the actions to work days, you can do
  - dayTime:
      onlyDays:
      - "monday"
      - "tuesday"
      - "wednesday"
      - "thursday"
      - "friday"
      startTime:
        hour: 18
        minute: 30
        second: 0
      endTime:
        hour: 22
        minute: 30
        second: 0

  # to pick a random sample of nodes/pods, you can specify either a size
  - randomSample:
      size: 1

  # this will pass all the nodes with the given probability,
  # or none otherwise
  - probability:
      probabilityPassAll: 1

  # The actions will be executed in the order specified
  actions:
  - stop:
      force: false
  - wait:
      seconds: 600
  - start:
  - wait:
      seconds: 1000
  - execute:
      cmd: "hostname"

I would like to use ip, az and name group properties.

Add prometheus /metrics support

The idea would be to be able to peek into what's going to give insights into the various measurables, like number of pods matched, filtered and actioned on, the number of machines taken up and down, etc.

Add python 2.7 support

A lot of people approached me to ask about the python 2.7 support.

If you need it for your setup, please vote for this issue.

Commands that take a subset of hosts return silently when the subset name is unknown

Commands that take a subset of hosts return silently when the subset name does not match any known string. For example, running docker thisdoesnotexist will return silently. This should be an explicit error if the string is not known as users are likely to accidentally forget the string and type something like docker info and then wonder why it returned without displaying anything.

interactive `kubectl` command should allow running locally

None of my cluster nodes have kubectl installed or configured on them. It would be nice to have a way for the kubectl commands to run on the workstation powerfulseal is running on. This lets you use the same auth / credentials as the cluster operator running powerfulseal. It also means you could potentially let unpriv users who might only have access to a certain namespace or something perform certain scenarios using this tool.

op is an undefined name

flake8 testing of https://github.com/bloomberg/powerfulseal on Python 3.6.3

$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics

./powerfulseal/cli/pscmd.py:334:16: F821 undefined name 'op'
            if op == "namespace":
               ^

https://travis-ci.org/bloomberg/powerfulseal/jobs/319560719#L612-L616

Running powerfulseal on AWS EKS

Hello,

I'm attempting to run powerfulseal against an EKS cluster. I have my AWS credentials exported as env variables before running and am running into an anonymous user issue.

$ seal -v interactive --aws --inventory-kubernetes --ssh-allow-missing-host-keys --kubeconfig REDACTED --ssh-path-to-private-key REDACTED --remote-user ec2-user

2019-03-04 13:25:11 REDACTED powerfulseal[11403] INFO Creating kubernetes client with config REDACTED
2019-03-04 13:25:11 REDACTED powerfulseal.k8s.k8s_client[11403] INFO Initializing with config: REDACTED
2019-03-04 13:25:11 REDACTED powerfulseal[11403] INFO Building AWS driver
2019-03-04 13:25:11 REDACTED powerfulseal[11403] INFO Attempting to read the inventory from kubernetes
2019-03-04 13:25:11 REDACTED powerfulseal.k8s.k8s_client[11403] ERROR (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '2cbe1482-0ad5-4964-b8a0-2c341c16f438', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 04 Mar 2019 18:25:11 GMT', 'Content-Length': '225'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"nodes is forbidden: User \"system:anonymous\" cannot list nodes at the cluster scope","reason":"Forbidden","details":{"kind":"nodes"},"code":403}

Need guidance to add Vmware as cloud option

Our application supports running on VMware and Openstack as platform, We need guidance on how we can add Vmware as cloud option.
We need to run chaos test against VMware as platform

Issue with running powerfulseal

I'm trying to run a powerful seal in virtual env and it's throwing me an error:

(env1) [ypoplavs@localhost]$ python -m powerfulseal.cli --kube-config path/to/config --inventory-kubernetes --interactive -v
Traceback (most recent call last):
File "/usr/local/lib/python3.4/runpy.py", line 170, in _run_module_as_main
"main", mod_spec)
File "/usr/local/lib/python3.4/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ypoplavs/env1/lib/python3.4/site-packages/powerfulseal/cli/main.py", line 198, in
start()
File "/home/ypoplavs/env1/lib/python3.4/site-packages/powerfulseal/cli/main.py", line 195, in start
main(sys.argv[1:])
File "/home/ypoplavs/env1/lib/python3.4/site-packages/powerfulseal/cli/main.py", line 135, in main
cloud=args.open_stack_cloud,
File "/home/ypoplavs/env1/lib/python3.4/site-packages/powerfulseal/clouddrivers/open_stack_driver.py", line 71, in init
self.conn = create_connection_from_config(cloud)
File "/home/ypoplavs/env1/lib/python3.4/site-packages/powerfulseal/clouddrivers/open_stack_driver.py", line 26, in create_connection_from_config
cloud = occ.get_one_cloud(name)
File "/home/ypoplavs/env1/lib/python3.4/site-packages/openstack/config/loader.py", line 1042, in get_one
auth_plugin = loader.load_from_options(**config['auth'])
File "/home/ypoplavs/env1/lib/python3.4/site-packages/keystoneauth1/loading/base.py", line 162, in load_from_options
raise exceptions.MissingRequiredOptions(missing_required)
keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url

It seems like an error is related to Openstack provider, however, I am using AWS. Should i specify that provider is AWS? However, I do not see it as an argument in --help.

Thank you in advance.

Push to PyPi

For an even quicker way of starting, it would be great to have a quick docker run command.

Add powerful seal logo

powerful-seal-logos.zip

Docker command

any reason why I'm getting state=NodeState.UNKNOWN
(seal) $ sync
ERROR:powerfulseal.clouddrivers.no_cloud_driver:Trying to sync things while using a no-cloud driver. If you don't expect to be seeing this, you might want to rethink some of your choices
[node no=30 id=fake-172.17.1.171 ip=172.17.1.171 az=nope groups=['amd64', 'kube-master-1.onprem.domain.com', 'linux', 'true'] name=local-172.17.1.171 state=NodeState.UNKNOWN]

not able to run powerfulseal on master node with python 3

even after installing python 3.5 and created python virtual env, still powerfulSeal use python 2.7?

(my_env) :/software/environmentsPython3$ python pwseal.py
Traceback (most recent call last):
File "/home/luser/.local/bin/powerfulseal", line 11, in
sys.exit(start())
File "/home/luser/.local/lib/python2.7/site-packages/powerfulseal/cli/main.py", line 224, in start
main(sys.argv[1:])
File "/home/luser/.local/lib/python2.7/site-packages/powerfulseal/cli/main.py", line 169, in main
k8s_client = K8sClient(kube_config=kube_config)
File "/home/luser/.local/lib/python2.7/site-packages/powerfulseal/k8s/k8s_client.py", line 29, in init
kubernetes.config.load_kube_config(config_file=kube_config)
File "/home/luser/.local/lib/python2.7/site-packages/kubernetes/config/kube_config.py", line 300, in load_kube_config
client_configuration=client_configuration).load_and_set()
File "/home/luser/.local/lib/python2.7/site-packages/kubernetes/config/kube_config.py", line 206, in load_and_set
self._load_cluster_info()
File "/home/luser/.local/lib/python2.7/site-packages/kubernetes/config/kube_config.py", line 185, in _load_cluster_info
file_base_path=self._config_base_path).as_file()
File "/home/luser/.local/lib/python2.7/site-packages/kubernetes/config/kube_config.py", line 82, in as_file
base64.decodestring(self._data.encode()))
File "/home/luser/.local/lib/python2.7/site-packages/kubernetes/config/kube_config.py", line 53, in _create_temp_file_with_content
fd.write(content.encode() if isinstance(content, str) else content)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc0 in position 2: ordinal not in range(128)
(my_env) :/software/environmentsPython3$ python -V
Python 3.5.2

Any detailed documentation?

I want to use powerfulseal in our k8s cluster, is there any detailed documentation?
When I installed powerfulseal with pip install powerfulseal, I encountered the following problem：

 ✘ zhanghui@慧儿 ~ powerfulseal --help
command not found: powerfulseal

Installation information is as follows：

Installing collected packages: prometheus-client, monotonic, humanfriendly, coloredlogs, pycparser, cffi, pynacl, asn1crypto, idna, enum34, ipaddress, cryptography, bcrypt, paramiko, spur, decorator, dogpile.cache, pbr, iso8601, certifi, chardet, requests, stevedore, os-service-types, keystoneauth1, munch, requestsexceptions, netifaces, jsonpointer, jsonpatch, appdirs, openstacksdk, websocket-client, httplib2, pyasn1-modules, oauth2client, kubernetes, itsdangerous, Werkzeug, MarkupSafe, Jinja2, click, Flask, flask-cors, future, ConfigArgParse, botocore, boto3, termcolor, flask-swagger-ui, functools32, jsonschema, powerfulseal
  The script humanfriendly is installed in '/Users/zhanghui/Library/Python/2.7/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  The script coloredlogs is installed in '/Users/zhanghui/Library/Python/2.7/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  The script pbr is installed in '/Users/zhanghui/Library/Python/2.7/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  The script chardetect is installed in '/Users/zhanghui/Library/Python/2.7/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  The script openstack-inventory is installed in '/Users/zhanghui/Library/Python/2.7/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  The script flask is installed in '/Users/zhanghui/Library/Python/2.7/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  The scripts futurize and pasteurize are installed in '/Users/zhanghui/Library/Python/2.7/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  The script jsonschema is installed in '/Users/zhanghui/Library/Python/2.7/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  The scripts powerfulseal and seal are installed in '/Users/zhanghui/Library/Python/2.7/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed ConfigArgParse-0.13.0 Flask-0.12.4 Jinja2-2.10 MarkupSafe-1.1.0 Werkzeug-0.14.1 appdirs-1.4.3 asn1crypto-0.24.0 bcrypt-3.1.5 boto3-1.9.71 botocore-1.12.71 certifi-2018.11.29 cffi-1.11.5 chardet-3.0.4 click-7.0 coloredlogs-10.0 cryptography-2.4.2 decorator-4.3.0 dogpile.cache-0.6.8 enum34-1.1.6 flask-cors-3.0.7 flask-swagger-ui-3.18.0 functools32-3.2.3.post2 future-0.17.1 httplib2-0.12.0 humanfriendly-4.17 idna-2.8 ipaddress-1.0.22 iso8601-0.1.12 itsdangerous-1.1.0 jsonpatch-1.23 jsonpointer-2.0 jsonschema-2.6.0 keystoneauth1-3.11.2 kubernetes-1.0.2 monotonic-1.5 munch-2.3.2 netifaces-0.10.7 oauth2client-4.1.3 openstacksdk-0.22.0 os-service-types-1.4.0 paramiko-2.4.2 pbr-5.1.1 powerfulseal-2.0.1 prometheus-client-0.3.1 pyasn1-modules-0.2.2 pycparser-2.19 pynacl-1.3.0 requests-2.21.0 requestsexceptions-1.4.0 spur-0.3.20 stevedore-1.30.0 termcolor-1.1.0 websocket-client-0.54.0

Fail to install powerfulseal because of locale?

ubuntu 16.04
python 2.7.12

root@i-mki26uqf:/opt# python --version  
Python 2.7.12

root@i-mki26uqf:/opt# pip install powerfulseal  
Traceback (most recent call last):  
  File "/usr/bin/pip", line 11, in <module>  
    sys.exit(main())  
  File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 215, in main  
    locale.setlocale(locale.LC_ALL, '')  
  File "/usr/lib/python2.7/locale.py", line 581, in setlocale  
    return _setlocale(category, locale)  
locale.Error: unsupported locale setting

powerfullseal wont run

Hi
Ive installed powerfullseal but when i run it it gives me error

powerfulseal --help
Traceback (most recent call last):
  File "/usr/local/bin/powerfulseal", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3142, in <module>
    @_call_aside
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3126, in _call_aside
    f(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3155, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 666, in _build_master
    return cls._build_from_requirements(__requires__)
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 679, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 872, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (pyasn1 0.3.7 (/usr/local/lib/python2.7/site-packages), Requirement.parse('pyasn1<0.5.0,>=0.4.1'), set(['pyasn1-modules']))

My python is 2.7.13
Ive tried it on virtualenv , tried to install master branch.
Nothing works.

unable to run powerfulseal

I am trying to setup powerfulseal for testing of our kubernetes cluster. Seeing following error message when I run version command. I did install powerfulseal with pip install

powerfulseal --version
Traceback (most recent call last):
File "/usr/bin/powerfulseal", line 11, in
load_entry_point('powerfulseal==1.1.1', 'console_scripts', 'powerfulseal')()
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 587, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 2800, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 2431, in load
return self.resolve()
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 2437, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "/usr/lib/python2.7/site-packages/powerfulseal/cli/main.py", line 26, in
from ..clouddrivers import OpenStackDriver
File "/usr/lib/python2.7/site-packages/powerfulseal/clouddrivers/init.py", line 17, in
from .driver import AbstractDriver
File "/usr/lib/python2.7/site-packages/powerfulseal/clouddrivers/driver.py", line 20
class AbstractDriver(metaclass=abc.ABCMeta):

kubectl delete pod in pod_scenario's action_kill

In a situation where the nodes are running in private subnets and we do not have ssh access to the nodes, would it make sense to randomly delete pods (kubectl delete pods) in a given namespace as part of the pod_scenario's action_kill rather then jumping on the node and killing the container ?

AWS Driver Does Not Return Public IP of Node

Given:
running kubernetes on aws

Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T19:44:19Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

Powerfulseal version. (no version command...that should be added...)

Running powerfulseal external to the target kubernetes cluster

Issues:
Unable to get an accessible IP for each Node to perform the desired actions.

Analysis:
The node object does not have a field to capture a public IP (or external IP). It only contains the internal cluster IP.
The AWS cloud driver does get the public_ip, but the implementation does not return that IP when matching a pod's IP address (which is usually the internal IP address).
The nodes internal IP address is not accessible from outside the kubernetes cluster.

Proposed Solution:

add and external IP field to the node object
update the cloud drivers to populate that field correctly (if available)
update cloud drivers (AWS in particular) to return the matching node (including the external IP field)
update node_inventory to also find nodes based on external IP
update tests to test this added node field.

I have a working version and will PR it back, if you find the proposal acceptable.

Adding Reviewers To This Project

Is there a procedure for adding reviewers or requesting to be added as a reviewer/owener to this project?

Seal crashes on listing pods in a namespace

I have some pods in a namespace stresstest:

$ kubectl get pods -n stresstest
NAME                       READY     STATUS    RESTARTS   AGE
mirrorer-bff584d74-ttcnz   1/1       Running   0          2d
mirrorer-bff584d74-vbs7l   1/1       Running   4          2d
v3-2-5-bdc8d9858-gptff     2/2       Running   0          2h

However attempting to retrieve the set of pods in my namespace seems to cause the seal to die and untimely death:

$ powerfulseal --inventory-kubernetes --interactive --kube-config /kubernetes/conf/kubeconfig
(seal) $ help pods

        List pods
        Syntax:
            pods namespace [selector]
        Selector is in the kubernetes native form: app=something,ver=1
        
(seal) $ pods stresstest 
Traceback (most recent call last):
  File "/home/cloud-user/powerfulseal/env/bin/powerfulseal", line 9, in <module>
    load_entry_point('powerfulseal==1.0.1', 'console_scripts', 'powerfulseal')()
  File "/home/cloud-user/powerfulseal/powerfulseal/cli/__main__.py", line 195, in start
    main(sys.argv[1:])
  File "/home/cloud-user/powerfulseal/powerfulseal/cli/__main__.py", line 178, in main
    cmd.cmdloop()
  File "/opt/rh/rh-python35/root/usr/lib64/python3.5/cmd.py", line 138, in cmdloop
    stop = self.onecmd(line)
  File "/opt/rh/rh-python35/root/usr/lib64/python3.5/cmd.py", line 217, in onecmd
    return func(arg)
  File "/home/cloud-user/powerfulseal/powerfulseal/cli/pscmd.py", line 346, in do_pods
    selector=selector
  File "/home/cloud-user/powerfulseal/powerfulseal/k8s/k8s_inventory.py", line 92, in find_pods
    ) for i, item in enumerate(pods)
  File "/home/cloud-user/powerfulseal/powerfulseal/k8s/k8s_inventory.py", line 92, in <listcomp>
    ) for i, item in enumerate(pods)
TypeError: 'NoneType' object is not iterable

I find that pods works fine in the default namespace but crashes even for the kube-system namespace.

def create_node_from_server(server, ip):
    """ Translate AWS EC2 Instance representation into a Node object.
    """
    return Node(
        id=server.id,
        ip=ip,
        az=server.placement['AvailabilityZone'],
        name="",
        state=server_status_to_state(server.state),
)

Error: (removed actual values, note that name doesn't have a value, and also there is no field uid)

Filtered set length: 1
Acting on these: [[node no=xxx id=<ID> ip=xx.xx.xx.xx az=ZONE groups=['ARCH', 'AZ', 'INSTSANCE_TYPE', 'NODE_LABEL', 'ip-xx-xx-xx-xx', 'OS', 'node'] name= state=NodeState.UP]]
Action stop on [node no=xxx id=<ID> ip=xx.xx.xx.xx az=ZONE groups=['ARCH', 'AZ', 'INSTSANCE_TYPE', 'NODE_LABEL', 'ip-xx-xx-xx-xx', 'OS', 'node'] name= state=NodeState.UP]
Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/opt-zvmYt2-H/src/powerfulseal/powerfulseal/policy/node_scenario.py", line 61, in action_stop
    self.metric_collector.add_node_stopped_metric(item)
  File "/root/.local/share/virtualenvs/opt-zvmYt2-H/src/powerfulseal/powerfulseal/metriccollectors/prometheus_collector.py", line 73, in add_node_stopped_metric
    NODE_STOPS.labels(STATUS_SUCCESS, node.uid, node.name).inc()
AttributeError: 'Node' object has no attribute 'uid'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/opt-zvmYt2-H/bin/powerfulseal", line 11, in <module>
    load_entry_point('powerfulseal', 'console_scripts', 'powerfulseal')()
  File "/root/.local/share/virtualenvs/opt-zvmYt2-H/src/powerfulseal/powerfulseal/cli/__main__.py", line 391, in start
    main(sys.argv[1:])
  File "/root/.local/share/virtualenvs/opt-zvmYt2-H/src/powerfulseal/powerfulseal/cli/__main__.py", line 387, in main
    metric_collector=metric_collector)
  File "/root/.local/share/virtualenvs/opt-zvmYt2-H/src/powerfulseal/powerfulseal/policy/policy_runner.py", line 88, in run
    scenario.execute()
  File "/root/.local/share/virtualenvs/opt-zvmYt2-H/src/powerfulseal/powerfulseal/policy/scenario.py", line 65, in execute
    self.act(filtered_set)
  File "/root/.local/share/virtualenvs/opt-zvmYt2-H/src/powerfulseal/powerfulseal/policy/node_scenario.py", line 90, in act
    return self.act_mapping(items, actions, mapping)
  File "/root/.local/share/virtualenvs/opt-zvmYt2-H/src/powerfulseal/powerfulseal/policy/scenario.py", line 221, in act_mapping
    method(item, params)
  File "/root/.local/share/virtualenvs/opt-zvmYt2-H/src/powerfulseal/powerfulseal/policy/node_scenario.py", line 64, in action_stop
    self.metric_collector.add_node_stop_failed_metric(item)
  File "/root/.local/share/virtualenvs/opt-zvmYt2-H/src/powerfulseal/powerfulseal/metriccollectors/prometheus_collector.py", line 76, in add_node_stop_failed_metric
    NODE_STOPS.labels(STATUS_FAILURE, node.uid, node.name).inc()

Pipefile:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[packages]
powerfulseal = {git = "git://github.com/bloomberg/powerfulseal.git", editable = true, ref = "master"}
Pipfile = "*"

[dev-packages]

[requires]
python_version = "3.6"

Start command:

pipenv run powerfulseal --inventory-kubernetes --aws-cloud --kube-config ~/.kube/config  --prometheus-collector --prometheus-host '0.0.0.0' --prometheus-port 8080  --remote-user core --ssh-allow-missing-host-keys --verbose -v --run-policy-file chaos_policy.yml

If prometheus is not enabled the app works fine