powerfulseal / powerfulseal Goto Github PK
View Code? Open in Web Editor NEWA powerful testing tool for Kubernetes clusters.
License: Apache License 2.0
A powerful testing tool for Kubernetes clusters.
License: Apache License 2.0
Hello,
I'm attempting to run powerfulseal against an EKS cluster. I have my AWS credentials exported as env variables before running and am running into an anonymous user issue.
$ seal -v interactive --aws --inventory-kubernetes --ssh-allow-missing-host-keys --kubeconfig REDACTED --ssh-path-to-private-key REDACTED --remote-user ec2-user
2019-03-04 13:25:11 REDACTED powerfulseal[11403] INFO Creating kubernetes client with config REDACTED
2019-03-04 13:25:11 REDACTED powerfulseal.k8s.k8s_client[11403] INFO Initializing with config: REDACTED
2019-03-04 13:25:11 REDACTED powerfulseal[11403] INFO Building AWS driver
2019-03-04 13:25:11 REDACTED powerfulseal[11403] INFO Attempting to read the inventory from kubernetes
2019-03-04 13:25:11 REDACTED powerfulseal.k8s.k8s_client[11403] ERROR (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Audit-Id': '2cbe1482-0ad5-4964-b8a0-2c341c16f438', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon, 04 Mar 2019 18:25:11 GMT', 'Content-Length': '225'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"nodes is forbidden: User \"system:anonymous\" cannot list nodes at the cluster scope","reason":"Forbidden","details":{"kind":"nodes"},"code":403}
The idea would be to be able to peek into what's going to give insights into the various measurables, like number of pods matched, filtered and actioned on, the number of machines taken up and down, etc.
Hi
Ive installed powerfullseal but when i run it it gives me error
powerfulseal --help
Traceback (most recent call last):
File "/usr/local/bin/powerfulseal", line 6, in <module>
from pkg_resources import load_entry_point
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3142, in <module>
@_call_aside
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3126, in _call_aside
f(*args, **kwargs)
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 3155, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 666, in _build_master
return cls._build_from_requirements(__requires__)
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 679, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/usr/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 872, in resolve
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (pyasn1 0.3.7 (/usr/local/lib/python2.7/site-packages), Requirement.parse('pyasn1<0.5.0,>=0.4.1'), set(['pyasn1-modules']))
My python is 2.7.13
Ive tried it on virtualenv , tried to install master branch.
Nothing works.
None of my cluster nodes have kubectl installed or configured on them. It would be nice to have a way for the kubectl commands to run on the workstation powerfulseal is running on. This lets you use the same auth / credentials as the cluster operator running powerfulseal. It also means you could potentially let unpriv users who might only have access to a certain namespace or something perform certain scenarios using this tool.
Whether to support?
The property "percentage" seems to expect a value between 0 and 1 while the name suggests a value between 0 and 100.
I think it would be clearer to use if the property was named "proportion"/"ratio" or if the scale of the value was changed.
In a situation where the nodes are running in private subnets
and we do not have ssh
access to the nodes, would it make sense to randomly delete pods (kubectl delete pods
) in a given namespace as part of the pod_scenario
's action_kill
rather then jumping on the node and killing the container ?
The cli should have a command to show the current version of powerfulseal.
This also implies maintaining a semver version value somewhere that corresponds with releases etc.
Is there a procedure for adding reviewers or requesting to be added as a reviewer/owener to this project?
Hi,
need more explanation on how this works in total?
my expectation was after setting below in policy file powerfulseal will start killing my pods once the start time hits and will run for 24 hours. but that never the case it start with saying too early and then continuously shows too late, why?
current time was :- 12: 01
The day of week :- Monday
My current configuration in policy file as below -
- dayTime:
onlyDays:
- "monday"
- "tuesday"
startTime:
hour: 12
minute: 03
second: 00
endTime:
hour: 11
minute: 50
second: 00
From logs -
INFO:powerfulseal.cli.main:Attempting to read the inventory from kubernetes
ERROR:powerfulseal.clouddrivers.no_cloud_driver:Trying to sync things while using a no-cloud driver. If you don't expect to be seeing this, you might want to rethink some of your choices
INFO:powerfulseal.policy.scenario.The Chaos Testing:Matched 1 pods in namespace powerfulsealtest
INFO:powerfulseal.policy.scenario.The Chaos Testing:Matching [pod #0 name=sise-748bf8d98b-w29qk namespace=powerfulsealtest containers=1 ip= host_ip= state=Running labels:owner=luser,pod-template-hash=3046948546,run=sise]
INFO:powerfulseal.policy.scenario.The Chaos Testing:Initial set length: 1
INFO:powerfulseal.policy.scenario.The Chaos Testing:Filter property: 1 -> 1 items
INFO:powerfulseal.policy.scenario.The Chaos Testing:Filter property: 1 -> 1 items
INFO:powerfulseal.policy.scenario.The Chaos Testing:Now is datetime.datetime(2018, 9, 3, 12, 2, 9, 10619)
### INFO:powerfulseal.policy.scenario.The Chaos Testing:Too early
INFO:powerfulseal.policy.scenario.The Chaos Testing:Filter dayTime: 1 -> 0 items
INFO:powerfulseal.policy.scenario.The Chaos Testing:Empty set after {'dayTime': {'onlyDays': ['monday', 'tuesday'], 'startTime': {'second': 0, 'hour': 12, 'minute': 3}, 'endTime': {'second': 0, 'hour': 11, 'minute': 50}}}
Filtered to empty set
INFO:powerfulseal.policy.scenario.The Chaos Testing:Filtered set length: 0
INFO:powerfulseal.policy.scenario.The Chaos Testing:Done
INFO:powerfulseal.policy.policy_runner:Sleeping for 177 seconds
ERROR:powerfulseal.clouddrivers.no_cloud_driver:Trying to sync things while using a no-cloud driver. If you don't expect to be seeing this, you might want to rethink some of your choices
INFO:powerfulseal.policy.scenario.The Chaos Testing:Matched 1 pods in namespace powerfulsealtest
INFO:powerfulseal.policy.scenario.The Chaos Testing:Matching [pod #0 name=sise-748bf8d98b-w29qk namespace=powerfulsealtest containers=1 ip= host_ip= state=Running labels:owner=,pod-template-hash=3046948546,run=sise]
INFO:powerfulseal.policy.scenario.The Chaos Testing:Initial set length: 1
INFO:powerfulseal.policy.scenario.The Chaos Testing:Filter property: 1 -> 1 items
INFO:powerfulseal.policy.scenario.The Chaos Testing:Filter property: 1 -> 1 items
INFO:powerfulseal.policy.scenario.The Chaos Testing:Now is datetime.datetime(2018, 9, 3, 12, 3, 23, 774632)
INFO:powerfulseal.policy.scenario.The Chaos Testing:Filter dayTime: 1 -> 0 items
INFO:powerfulseal.policy.scenario.The Chaos Testing:Empty set after {'dayTime': {'onlyDays': ['monday', 'tuesday'], 'startTime': {'second': 0, 'hour': 12, 'minute': 3}, 'endTime': {'second': 0, 'hour': 11, 'minute': 50}}}
Filtered to empty set
INFO:powerfulseal.policy.scenario.The Chaos Testing:Filtered set length: 0
INFO:powerfulseal.policy.scenario.The Chaos Testing:Done
INFO:powerfulseal.policy.policy_runner:Sleeping for 301 seconds
On my Oracle Linux 7.3 box with Python 2.7.5, I had to perform some additional steps to get powerfulseal to work.
$ sudo pip install powerfulseal
Installing collected packages: ConfigArgParse, click, MarkupSafe, Jinja2, Werkzeug, itsdangerous, Flask, termcolor, pyparsing, packaging, deprecation, iso8601, pbr, stevedore, keystoneauth1, os-service-types, munch, netifaces, futures, appdirs, requestsexceptions, jmespath, jsonpointer, jsonpatch, dogpile.cache, openstacksdk, bcrypt, asn1crypto, cryptography, pynacl, paramiko, spur, python-dateutil, pyasn1-modules, rsa, httplib2, oauth2client, kubernetes, powerfulseal
Running setup.py install for ConfigArgParse ... done
Running setup.py install for MarkupSafe ... done
Running setup.py install for itsdangerous ... done
Running setup.py install for termcolor ... done
Found existing installation: pyparsing 1.5.6 DEPRECATION: Uninstalling a distutils installed project (pyparsing) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
Uninstalling pyparsing-1.5.6: Successfully uninstalled pyparsing-1.5.6
Running setup.py install for deprecation ... done
Running setup.py install for munch ... done
Running setup.py install for netifaces ... done
Running setup.py install for dogpile.cache ... done
Found existing installation: cryptography 1.3.1 Uninstalling cryptography-1.3.1: Successfully uninstalled cryptography-1.3.1
Running setup.py install for httplib2 ... done
Running setup.py install for powerfulseal ... done
Successfully installed ConfigArgParse-0.13.0 Flask-0.12.2 Jinja2-2.10 MarkupSafe-1.0 Werkzeug-0.14.1 appdirs-1.4.3 asn1crypto-0.24.0 bcrypt-3.1.4 click-6.7 cryptography-2.2.2 deprecation-2.0.2 dogpile.cache-0.6.5 futures-3.2.0 httplib2-0.11.3 iso8601-0.1.12 itsdangerous-0.24 jmespath-0.9.3 jsonpatch-1.23 jsonpointer-2.0 keystoneauth1-3.5.0 kubernetes-1.0.2 munch-2.3.1 netifaces-0.10.6 oauth2client-4.1.2 openstacksdk-0.12.0 os-service-types-1.2.0 packaging-17.1 paramiko-2.4.1 pbr-4.0.2 powerfulseal-1.1.1 pyasn1-modules-0.2.1 pynacl-1.2.1 pyparsing-2.2.0 python-dateutil-2.7.2 requestsexceptions-1.4.0 rsa-3.4.2 spur-0.3.20 stevedore-1.28.0 termcolor-1.1.0
after I completed the installation for powerfulseal (as per logs above), I wasn't able to execute powerfulseal.
In order to resolve the issues, these are the steps that I had to perform -
sudo pip install pyasn1 --upgrade
sudo pip install cffi --upgrade
sudo pip install configparser
File: "/usr/lib/python2.7/site-packages/powerfulseal/cli/pscmd.py" - Added the line below to the top of the file.
from __future__ import print_function
File: /usr/lib/python2.7/site-packages/powerfulseal/clouddrivers/driver.py - Added the first two lines to import six for compatibility and updated the code as shown in line 3 below.
import abc, six
@six.add_metaclass(abc.ABCMeta)
class AbstractDriver():
ubuntu 16.04
python 2.7.12
root@i-mki26uqf:/opt# python --version
Python 2.7.12
root@i-mki26uqf:/opt# pip install powerfulseal
Traceback (most recent call last):
File "/usr/bin/pip", line 11, in <module>
sys.exit(main())
File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 215, in main
locale.setlocale(locale.LC_ALL, '')
File "/usr/lib/python2.7/locale.py", line 581, in setlocale
return _setlocale(category, locale)
locale.Error: unsupported locale setting
Currently, to run the PowerfulSeal
, it is necessary to provide cloud provider details.
The seal could be more useful if it had an extra mode for constricted environments, in which it only speaks to Kubernetes
, and/or Kubernetes
+ SSH
. This way, it could provide more value in the corner cases.
With the current implementation of spur lib for ssh connectivity, I do not see a way for providing an ssh client config (whereby one may have a client config that uses a ProxyCommand configuration). Please consider supporting ssh client config (rather than passing a path to a private key), as modern access security models support multiple keys for access to inside hosts through bastion/edge nodes.
How would you envision using powerfulseal for a bare metal on premise kubernetes environment? All of the servers have IPMI access for remote power. Would it be a new cloud driver, or would it be an addition to #17 along with some ssh magic? I might be able to work on some of it with an idea of how this should be implemented with the idea of getting it back upstream.
Hello Team,
I have a project in GCP and have Kubernetes cluster in it where in for my testing purpose I am planning to kill my pods/nodes using Powerfulseal. Can you please let me know if Powerfulseal supports with GCP/GKE.
Can I point seal at a standard KUBE_CONFIG to access a baremetal cluster and kill pods?
Because it is more better-er.
even after installing python 3.5 and created python virtual env, still powerfulSeal use python 2.7?
(my_env) :/software/environmentsPython3$ python pwseal.py/software/environmentsPython3$ python -V
Traceback (most recent call last):
File "/home/luser/.local/bin/powerfulseal", line 11, in
sys.exit(start())
File "/home/luser/.local/lib/python2.7/site-packages/powerfulseal/cli/main.py", line 224, in start
main(sys.argv[1:])
File "/home/luser/.local/lib/python2.7/site-packages/powerfulseal/cli/main.py", line 169, in main
k8s_client = K8sClient(kube_config=kube_config)
File "/home/luser/.local/lib/python2.7/site-packages/powerfulseal/k8s/k8s_client.py", line 29, in init
kubernetes.config.load_kube_config(config_file=kube_config)
File "/home/luser/.local/lib/python2.7/site-packages/kubernetes/config/kube_config.py", line 300, in load_kube_config
client_configuration=client_configuration).load_and_set()
File "/home/luser/.local/lib/python2.7/site-packages/kubernetes/config/kube_config.py", line 206, in load_and_set
self._load_cluster_info()
File "/home/luser/.local/lib/python2.7/site-packages/kubernetes/config/kube_config.py", line 185, in _load_cluster_info
file_base_path=self._config_base_path).as_file()
File "/home/luser/.local/lib/python2.7/site-packages/kubernetes/config/kube_config.py", line 82, in as_file
base64.decodestring(self._data.encode()))
File "/home/luser/.local/lib/python2.7/site-packages/kubernetes/config/kube_config.py", line 53, in _create_temp_file_with_content
fd.write(content.encode() if isinstance(content, str) else content)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc0 in position 2: ordinal not in range(128)
(my_env) :
Python 3.5.2
Hi Team,
Will Powerful seal works for the open shift origin ?
Thanks
Every time when the app tries to stop a node it will crash in prometheus collector, because object Node
has no attribute uid
.
It seems that pod
object has this attribute, but not node
:
https://github.com/bloomberg/powerfulseal/blob/f3014fa75477b1f123e87fa12df8e76491e3cd2a/powerfulseal/clouddrivers/aws_driver.py#L30-L39:
def create_node_from_server(server, ip):
""" Translate AWS EC2 Instance representation into a Node object.
"""
return Node(
id=server.id,
ip=ip,
az=server.placement['AvailabilityZone'],
name="",
state=server_status_to_state(server.state),
)
Error: (removed actual values, note that name
doesn't have a value, and also there is no field uid
)
Filtered set length: 1
Acting on these: [[node no=xxx id=<ID> ip=xx.xx.xx.xx az=ZONE groups=['ARCH', 'AZ', 'INSTSANCE_TYPE', 'NODE_LABEL', 'ip-xx-xx-xx-xx', 'OS', 'node'] name= state=NodeState.UP]]
Action stop on [node no=xxx id=<ID> ip=xx.xx.xx.xx az=ZONE groups=['ARCH', 'AZ', 'INSTSANCE_TYPE', 'NODE_LABEL', 'ip-xx-xx-xx-xx', 'OS', 'node'] name= state=NodeState.UP]
Traceback (most recent call last):
File "/root/.local/share/virtualenvs/opt-zvmYt2-H/src/powerfulseal/powerfulseal/policy/node_scenario.py", line 61, in action_stop
self.metric_collector.add_node_stopped_metric(item)
File "/root/.local/share/virtualenvs/opt-zvmYt2-H/src/powerfulseal/powerfulseal/metriccollectors/prometheus_collector.py", line 73, in add_node_stopped_metric
NODE_STOPS.labels(STATUS_SUCCESS, node.uid, node.name).inc()
AttributeError: 'Node' object has no attribute 'uid'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/.local/share/virtualenvs/opt-zvmYt2-H/bin/powerfulseal", line 11, in <module>
load_entry_point('powerfulseal', 'console_scripts', 'powerfulseal')()
File "/root/.local/share/virtualenvs/opt-zvmYt2-H/src/powerfulseal/powerfulseal/cli/__main__.py", line 391, in start
main(sys.argv[1:])
File "/root/.local/share/virtualenvs/opt-zvmYt2-H/src/powerfulseal/powerfulseal/cli/__main__.py", line 387, in main
metric_collector=metric_collector)
File "/root/.local/share/virtualenvs/opt-zvmYt2-H/src/powerfulseal/powerfulseal/policy/policy_runner.py", line 88, in run
scenario.execute()
File "/root/.local/share/virtualenvs/opt-zvmYt2-H/src/powerfulseal/powerfulseal/policy/scenario.py", line 65, in execute
self.act(filtered_set)
File "/root/.local/share/virtualenvs/opt-zvmYt2-H/src/powerfulseal/powerfulseal/policy/node_scenario.py", line 90, in act
return self.act_mapping(items, actions, mapping)
File "/root/.local/share/virtualenvs/opt-zvmYt2-H/src/powerfulseal/powerfulseal/policy/scenario.py", line 221, in act_mapping
method(item, params)
File "/root/.local/share/virtualenvs/opt-zvmYt2-H/src/powerfulseal/powerfulseal/policy/node_scenario.py", line 64, in action_stop
self.metric_collector.add_node_stop_failed_metric(item)
File "/root/.local/share/virtualenvs/opt-zvmYt2-H/src/powerfulseal/powerfulseal/metriccollectors/prometheus_collector.py", line 76, in add_node_stop_failed_metric
NODE_STOPS.labels(STATUS_FAILURE, node.uid, node.name).inc()
Pipefile:
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"
[packages]
powerfulseal = {git = "git://github.com/bloomberg/powerfulseal.git", editable = true, ref = "master"}
Pipfile = "*"
[dev-packages]
[requires]
python_version = "3.6"
Start command:
pipenv run powerfulseal --inventory-kubernetes --aws-cloud --kube-config ~/.kube/config --prometheus-collector --prometheus-host '0.0.0.0' --prometheus-port 8080 --remote-user core --ssh-allow-missing-host-keys --verbose -v --run-policy-file chaos_policy.yml
If prometheus is not enabled the app works fine
Hi!
I wanted to try powerfulseal, but I found that after pip-installing it I cannot even run the help command. It seems there's a dependency issue in the python package (you may want to check the ci job of master, it should be failing).
I noticed os_client_config
is now embedded into openstack
package, so a simple patch like the following would do.
diff --git a/powerfulseal/clouddrivers/open_stack_driver.py b/powerfulseal/clouddrivers/open_stack_driver.py
index 71708a1..c343f08 100644
--- a/powerfulseal/clouddrivers/open_stack_driver.py
+++ b/powerfulseal/clouddrivers/open_stack_driver.py
@@ -15,15 +15,14 @@
import logging
-import os_client_config
-from openstack import connection
from . import AbstractDriver
from ..node import Node, NodeState
def create_connection_from_config(name=None):
""" Creates a new open stack connection """
- occ = os_client_config.OpenStackConfig()
+ from openstack import connection, OpenStackConfig
+ occ = OpenStackConfig()
cloud = occ.get_one_cloud(name)
return connection.from_config(cloud_config=cloud)
@@ -106,4 +105,3 @@ class OpenStackDriver(AbstractDriver):
""" Delete a Node permanently.
"""
self.conn.compute.delete_server(node.id)
-
Hi,
Can you please provide me an example of an inventory file (apart from the one in tests) that allows me to match with group properties in the node_scenarios policy. I use this now,
[c]
52.58.80.238
[a]
3.120.248.11
[workers:children]
c
a
How do I provide more info here to match to the group properties for a policy like below?
config:
minSecondsBetweenRuns: 3600
maxSecondsBetweenRuns: 7200
# the scenarios describing actions on nodes
nodeScenarios:
# example of a policy using al the filters available
- name: "reboot control node"
# Choose the initial set of nodes to operate on.
# Note that this will be an union of all the notes you match (logical OR)
match:
- property:
name: "group"
value: "a"
# The filters are executed in the order specified and can be
# used multiple times, and piped from one to the next.
filters:
# property filters (all the property filters support regexp)
- property:
name: "group"
value: "a"
# time of execution filters
# to restrict the actions to work days, you can do
- dayTime:
onlyDays:
- "monday"
- "tuesday"
- "wednesday"
- "thursday"
- "friday"
startTime:
hour: 18
minute: 30
second: 0
endTime:
hour: 22
minute: 30
second: 0
# to pick a random sample of nodes/pods, you can specify either a size
- randomSample:
size: 1
# this will pass all the nodes with the given probability,
# or none otherwise
- probability:
probabilityPassAll: 1
# The actions will be executed in the order specified
actions:
- stop:
force: false
- wait:
seconds: 600
- start:
- wait:
seconds: 1000
- execute:
cmd: "hostname"
I would like to use ip, az and name group properties.
Scenario: multiple pod-specific matchers are used to match pods (namespace and deployment) and I wish to use a filter of the same criteria to effectively make the match an AND operation instead of OR
Currently there is no way to filter on these pod-specific matchers, being instead limited to property/dayTime/randomSample/probability (populated in the filter mapping in scenario.py
)
I'd like to propose adding the capability for pod_scenario and node_scenario to optionally add to the filter mapping to provide their own filters for cases that make sense.
Pod
and Node
classes should be implementing __hash__
with their respective IDs.
any reason why I'm getting state=NodeState.UNKNOWN
(seal) $ sync
ERROR:powerfulseal.clouddrivers.no_cloud_driver:Trying to sync things while using a no-cloud driver. If you don't expect to be seeing this, you might want to rethink some of your choices
[node no=30 id=fake-172.17.1.171 ip=172.17.1.171 az=nope groups=['amd64', 'kube-master-1.onprem.domain.com', 'linux', 'true'] name=local-172.17.1.171 state=NodeState.UNKNOWN]
Given:
running kubernetes on aws
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T19:44:19Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:00:57Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Powerfulseal version. (no version command...that should be added...)
Running powerfulseal external to the target kubernetes cluster
Issues:
Unable to get an accessible IP for each Node to perform the desired actions.
Analysis:
The node object does not have a field to capture a public IP (or external IP). It only contains the internal cluster IP.
The AWS cloud driver does get the public_ip, but the implementation does not return that IP when matching a pod's IP address (which is usually the internal IP address).
The nodes internal IP address is not accessible from outside the kubernetes cluster.
Proposed Solution:
I have a working version and will PR it back, if you find the proposal acceptable.
root@node1:~#seal --kube-config /root/kubeconfig --interactive --inventory-kubernetes --no-cloud
(seal) $ docker
Executing '['sh', '-c', 'sudo docker ']' on local-172.24.26.66
--------------------------------------------------------------------------------
172.24.26.66
Error creating SSH connection
Original error: Authentication failed.
Executing '['sh', '-c', 'sudo docker ']' on local-172.24.26.67
--------------------------------------------------------------------------------
172.24.26.67
Error creating SSH connection
Original error: Authentication failed.
Executing '['sh', '-c', 'sudo docker ']' on local-172.24.26.68
--------------------------------------------------------------------------------
172.24.26.68
Error creating SSH connection
Original error: Authentication failed.
Add an Azure Cloud Driver.
I'm trying to run a powerful seal in virtual env and it's throwing me an error:
(env1) [ypoplavs@localhost]$ python -m powerfulseal.cli --kube-config path/to/config --inventory-kubernetes --interactive -v
Traceback (most recent call last):
File "/usr/local/lib/python3.4/runpy.py", line 170, in _run_module_as_main
"main", mod_spec)
File "/usr/local/lib/python3.4/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/ypoplavs/env1/lib/python3.4/site-packages/powerfulseal/cli/main.py", line 198, in
start()
File "/home/ypoplavs/env1/lib/python3.4/site-packages/powerfulseal/cli/main.py", line 195, in start
main(sys.argv[1:])
File "/home/ypoplavs/env1/lib/python3.4/site-packages/powerfulseal/cli/main.py", line 135, in main
cloud=args.open_stack_cloud,
File "/home/ypoplavs/env1/lib/python3.4/site-packages/powerfulseal/clouddrivers/open_stack_driver.py", line 71, in init
self.conn = create_connection_from_config(cloud)
File "/home/ypoplavs/env1/lib/python3.4/site-packages/powerfulseal/clouddrivers/open_stack_driver.py", line 26, in create_connection_from_config
cloud = occ.get_one_cloud(name)
File "/home/ypoplavs/env1/lib/python3.4/site-packages/openstack/config/loader.py", line 1042, in get_one
auth_plugin = loader.load_from_options(**config['auth'])
File "/home/ypoplavs/env1/lib/python3.4/site-packages/keystoneauth1/loading/base.py", line 162, in load_from_options
raise exceptions.MissingRequiredOptions(missing_required)
keystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url
It seems like an error is related to Openstack provider, however, I am using AWS. Should i specify that provider is AWS? However, I do not see it as an argument in --help.
Thank you in advance.
For an even quicker way of starting, it would be great to have a quick docker run
command.
I have some pods in a namespace stresstest
:
$ kubectl get pods -n stresstest
NAME READY STATUS RESTARTS AGE
mirrorer-bff584d74-ttcnz 1/1 Running 0 2d
mirrorer-bff584d74-vbs7l 1/1 Running 4 2d
v3-2-5-bdc8d9858-gptff 2/2 Running 0 2h
However attempting to retrieve the set of pods in my namespace seems to cause the seal to die and untimely death:
$ powerfulseal --inventory-kubernetes --interactive --kube-config /kubernetes/conf/kubeconfig
(seal) $ help pods
List pods
Syntax:
pods namespace [selector]
Selector is in the kubernetes native form: app=something,ver=1
(seal) $ pods stresstest
Traceback (most recent call last):
File "/home/cloud-user/powerfulseal/env/bin/powerfulseal", line 9, in <module>
load_entry_point('powerfulseal==1.0.1', 'console_scripts', 'powerfulseal')()
File "/home/cloud-user/powerfulseal/powerfulseal/cli/__main__.py", line 195, in start
main(sys.argv[1:])
File "/home/cloud-user/powerfulseal/powerfulseal/cli/__main__.py", line 178, in main
cmd.cmdloop()
File "/opt/rh/rh-python35/root/usr/lib64/python3.5/cmd.py", line 138, in cmdloop
stop = self.onecmd(line)
File "/opt/rh/rh-python35/root/usr/lib64/python3.5/cmd.py", line 217, in onecmd
return func(arg)
File "/home/cloud-user/powerfulseal/powerfulseal/cli/pscmd.py", line 346, in do_pods
selector=selector
File "/home/cloud-user/powerfulseal/powerfulseal/k8s/k8s_inventory.py", line 92, in find_pods
) for i, item in enumerate(pods)
File "/home/cloud-user/powerfulseal/powerfulseal/k8s/k8s_inventory.py", line 92, in <listcomp>
) for i, item in enumerate(pods)
TypeError: 'NoneType' object is not iterable
I find that pods works fine in the default namespace but crashes even for the kube-system
namespace.
Commands that take a subset of hosts return silently when the subset name does not match any known string. For example, running docker thisdoesnotexist
will return silently. This should be an explicit error if the string is not known as users are likely to accidentally forget the string and type something like docker info
and then wonder why it returned without displaying anything.
In interactive mode, for certain commands, the powerful seal tries to ssh into a destination host to run the command. Using the --inventory-kubernetes
flag makes the seal try to guess the inventory from kubernetes but if a host has both a public IP address and a private IP address, it tends to find the public IP address and then have ssh complain about the IP address not being present in known_hosts.
powerfulseal --inventory-kubernetes --interactive --kube-config /kubernetes/conf/kubeconfig
docker
. We get an error of the form:Executing '['sh', '-c', 'sudo docker ']' on sk1-worker-1
--------------------------------------------------------------------------------
xxx.xxx.xxx.xxx
Error creating SSH connection
Original error: Server 'xxx.xxx.xxx.xxx' not found in known_hosts
Where xxx.xxx.xxx.xxx
is the public IP address of a host that is present in known hosts.
Hello, I installed powerfulseal by pip in Ubuntu VM that was installed k8s.I ran into interactive mode, but It returns nothing when I executed kill pod, Is there any wrong or missed step?
root@taco-aio:~# powerfulseal --interactive --inventory-kubernetes --kube-config .kube/config
(seal) $ groups
amd64
enabled
linux
taco-aio
true
(seal) $ namespaces
ceph
default
kube-public
kube-system
openstack
(seal) $ pods openstack application=nova
[pod #0 name=nova-api-metadata-58d96477bc-5m8pf namespace=openstack containers=1 ip=172.16.106.22 host_ip=192.168.122.190 state
=Running labels:application=nova,pod-template-hash=1485203367,component=metadata,release_group=taco-nova]
[pod #1 name=nova-api-osapi-658fdf8688-knx9r namespace=openstack containers=1 ip=172.16.106.24 host_ip=192.168.122.190 state=Ru
nning labels:application=nova,pod-template-hash=2149894244,component=os-api,release_group=taco-nova]
[pod #2 name=nova-bootstrap-p6gw8 namespace=openstack containers=1 ip=172.16.106.49 host_ip=192.168.122.190 state=Succeeded lab
els:application=nova,controller-uid=cb019bea-0bbf-11e8-bca1-525400d69f89,component=bootstrap,job-name=nova-bootstrap,release_gr
oup=taco-nova]
[pod #3 name=nova-cell-setup-nmjk8 namespace=openstack containers=1 ip=172.16.106.28 host_ip=192.168.122.190 state=Succeeded la
bels:application=nova,controller-uid=cb039bf3-0bbf-11e8-bca1-525400d69f89,component=cell-setup,job-name=nova-cell-setup,release
_group=taco-nova]
[pod #4 name=nova-compute-5b9sx namespace=openstack containers=2 ip=192.168.122.190 host_ip=192.168.122.190 state=Running label
s:application=nova,pod-template-generation=1,component=compute,controller-revision-hash=3918579215,release_group=taco-nova]
[pod #5 name=nova-conductor-6cd6689fb9-6554f namespace=openstack containers=1 ip=172.16.106.44 host_ip=192.168.122.190 state=Ru
nning labels:application=nova,pod-template-hash=2782245965,component=conductor,release_group=taco-nova]
[pod #6 name=nova-consoleauth-f44969c95-tc9mx namespace=openstack containers=1 ip=172.16.106.35 host_ip=192.168.122.190 state=R
unning labels:application=nova,pod-template-hash=900525751,component=consoleauth,release_group=taco-nova]
[pod #7 name=nova-db-init-vbwrw namespace=openstack containers=3 ip=172.16.106.3 host_ip=192.168.122.190 state=Succeeded labels
:application=nova,controller-uid=cb07cca8-0bbf-11e8-bca1-525400d69f89,component=db-init,job-name=nova-db-init,release_group=tac
o-nova]
[pod #8 name=nova-db-sync-llvqc namespace=openstack containers=1 ip=172.16.106.41 host_ip=192.168.122.190 state=Succeeded label
s:application=nova,controller-uid=cb0c9fe1-0bbf-11e8-bca1-525400d69f89,component=db-sync,job-name=nova-db-sync,release_group=ta
co-nova]
[pod #9 name=nova-ks-endpoints-29zkm namespace=openstack containers=3 ip=172.16.106.7 host_ip=192.168.122.190 state=Succeeded l
abels:application=nova,controller-uid=cb11dcb7-0bbf-11e8-bca1-525400d69f89,component=ks-endpoints,job-name=nova-ks-endpoints,re
lease_group=taco-nova]
[pod #10 name=nova-ks-service-n2kcn namespace=openstack containers=1 ip=172.16.106.2 host_ip=192.168.122.190 state=Succeeded la
bels:application=nova,controller-uid=cb3709d5-0bbf-11e8-bca1-525400d69f89,component=ks-service,job-name=nova-ks-service,release
_group=taco-nova]
[pod #11 name=nova-ks-user-9lj5m namespace=openstack containers=1 ip=172.16.106.51 host_ip=192.168.122.190 state=Succeeded labe
ls:application=nova,controller-uid=cb3b1e7c-0bbf-11e8-bca1-525400d69f89,component=ks-user,job-name=nova-ks-user,release_group=t
aco-nova]
[pod #12 name=nova-novncproxy-7db5fb96b9-xtwph namespace=openstack containers=1 ip=192.168.122.190 host_ip=192.168.122.190 stat
e=Running labels:application=nova,pod-template-hash=3861965265,component=novnc-proxy,release_group=taco-nova]
[pod #13 name=nova-placement-api-5b794b96f8-dfhr2 namespace=openstack containers=1 ip=172.16.106.10 host_ip=192.168.122.190 sta
te=Running labels:application=nova,pod-template-hash=1635065294,component=placement,release_group=taco-nova]
[pod #14 name=nova-scheduler-59b5999769-jnr2n namespace=openstack containers=1 ip=172.16.106.9 host_ip=192.168.122.190 state=Ru
nning labels:application=nova,pod-template-hash=1561555325,component=scheduler,release_group=taco-nova]
(seal) $ kill 14
Node not found
(seal) $
For certain commands in interactive mode, the powerful seal connects to the host via ssh. If the host is not present in your known_hosts
, the seal will fail to connect and throw an error of the form:
Executing '['sh', '-c', 'sudo docker ']' on sk1-worker-1
--------------------------------------------------------------------------------
xxx.xxx.xxx.xxx
Error creating SSH connection
Original error: Server 'xxx.xxx.xxx.xxx' not found in known_hosts
There is no way to make the seal either add the IP address to your known hosts or have it ignore the known hosts file altogether. We should consider having ssh use -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no
on an explicit command from the user.
I am trying to setup powerfulseal for testing of our kubernetes cluster. Seeing following error message when I run version command. I did install powerfulseal with pip install
powerfulseal --version
Traceback (most recent call last):
File "/usr/bin/powerfulseal", line 11, in
load_entry_point('powerfulseal==1.1.1', 'console_scripts', 'powerfulseal')()
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 587, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 2800, in load_entry_point
return ep.load()
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 2431, in load
return self.resolve()
File "/usr/lib/python2.7/site-packages/pkg_resources/init.py", line 2437, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "/usr/lib/python2.7/site-packages/powerfulseal/cli/main.py", line 26, in
from ..clouddrivers import OpenStackDriver
File "/usr/lib/python2.7/site-packages/powerfulseal/clouddrivers/init.py", line 17, in
from .driver import AbstractDriver
File "/usr/lib/python2.7/site-packages/powerfulseal/clouddrivers/driver.py", line 20
class AbstractDriver(metaclass=abc.ABCMeta):
Add capability to ssh to the kubernetes nodes through a bastion host.
I want to use powerfulseal
in our k8s cluster, is there any detailed documentation?
When I installed powerfulseal
with pip install powerfulseal
, I encountered the following problem:
✘ zhanghui@慧儿 ~ powerfulseal --help
command not found: powerfulseal
Installing collected packages: prometheus-client, monotonic, humanfriendly, coloredlogs, pycparser, cffi, pynacl, asn1crypto, idna, enum34, ipaddress, cryptography, bcrypt, paramiko, spur, decorator, dogpile.cache, pbr, iso8601, certifi, chardet, requests, stevedore, os-service-types, keystoneauth1, munch, requestsexceptions, netifaces, jsonpointer, jsonpatch, appdirs, openstacksdk, websocket-client, httplib2, pyasn1-modules, oauth2client, kubernetes, itsdangerous, Werkzeug, MarkupSafe, Jinja2, click, Flask, flask-cors, future, ConfigArgParse, botocore, boto3, termcolor, flask-swagger-ui, functools32, jsonschema, powerfulseal
The script humanfriendly is installed in '/Users/zhanghui/Library/Python/2.7/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
The script coloredlogs is installed in '/Users/zhanghui/Library/Python/2.7/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
The script pbr is installed in '/Users/zhanghui/Library/Python/2.7/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
The script chardetect is installed in '/Users/zhanghui/Library/Python/2.7/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
The script openstack-inventory is installed in '/Users/zhanghui/Library/Python/2.7/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
The script flask is installed in '/Users/zhanghui/Library/Python/2.7/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
The scripts futurize and pasteurize are installed in '/Users/zhanghui/Library/Python/2.7/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
The script jsonschema is installed in '/Users/zhanghui/Library/Python/2.7/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
The scripts powerfulseal and seal are installed in '/Users/zhanghui/Library/Python/2.7/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed ConfigArgParse-0.13.0 Flask-0.12.4 Jinja2-2.10 MarkupSafe-1.1.0 Werkzeug-0.14.1 appdirs-1.4.3 asn1crypto-0.24.0 bcrypt-3.1.5 boto3-1.9.71 botocore-1.12.71 certifi-2018.11.29 cffi-1.11.5 chardet-3.0.4 click-7.0 coloredlogs-10.0 cryptography-2.4.2 decorator-4.3.0 dogpile.cache-0.6.8 enum34-1.1.6 flask-cors-3.0.7 flask-swagger-ui-3.18.0 functools32-3.2.3.post2 future-0.17.1 httplib2-0.12.0 humanfriendly-4.17 idna-2.8 ipaddress-1.0.22 iso8601-0.1.12 itsdangerous-1.1.0 jsonpatch-1.23 jsonpointer-2.0 jsonschema-2.6.0 keystoneauth1-3.11.2 kubernetes-1.0.2 monotonic-1.5 munch-2.3.2 netifaces-0.10.7 oauth2client-4.1.3 openstacksdk-0.22.0 os-service-types-1.4.0 paramiko-2.4.2 pbr-5.1.1 powerfulseal-2.0.1 prometheus-client-0.3.1 pyasn1-modules-0.2.2 pycparser-2.19 pynacl-1.3.0 requests-2.21.0 requestsexceptions-1.4.0 spur-0.3.20 stevedore-1.30.0 termcolor-1.1.0 websocket-client-0.54.0
I'm getting an error message when running this command bellow: I'm using Openshift which shouldn't really matter.
powerfulseal --run-policy-file example.policy --kube-config ~/.kube/config --inventory-kubernetes --no-cloud -v
ERROR:powerfulseal.clouddrivers.no_cloud_driver:Trying to sync things while using a no-cloud driver. If you don't expect to be seeing this, you might want to rethink some of your choices
Executing '['sh', '-c', 'sudo docker kill -s SIGKILL bb549a611670cc65322bacb4b674c11f0dd9e5c69732e13c93999d4860093206']' on local-10.0.2.15
Pod killed - namespace: metrics - name: alertmanager-metrics-0
Executing '['sh', '-c', 'sudo docker kill -s SIGKILL 6dd2bb3613fbf805ac3ae1e0775c341b500c7458f88319282ae7cced1cd06875']' on local-10.0.2.15
Pod killed - namespace: metrics - name: kube-state-metrics-6b67c7b958-6ccr2
Executing '['sh', '-c', 'sudo docker kill -s SIGKILL cb569d3e3f12c80fc7ad8d25ac97b80244a893d9dd029b7be73eadfb4d778233']' on local-10.0.2.15
Pod killed - namespace: metrics - name: node-exporter-stg5k
Executing '['sh', '-c', 'sudo docker kill -s SIGKILL 39798cab85e96f30af3f4c79bf83bb2c7591650b8e5eba354feb4f984eec704a']' on local-10.0.2.15
Pod killed - namespace: metrics - name: kubernetes-pod-chaos-monkey-b9879fb44-jw4mg
Executing '['sh', '-c', 'sudo docker kill -s SIGKILL 2ad332b2cae785608ed24d8a0bbc5b34240366b60091126ce2e0600bf1dc0564']' on local-10.0.2.15
My Policy is very simple:
podScenarios:
- name: "delete random pods"
match:
- namespace:
name: "metrics"
filters:
- property:
name: "state"
value: "Running"
actions:
- kill:
probability: 1
force: true
- wait:
seconds: 5
For instance, this should be allowed to be executed:
$ seal --validate-policy-file ./example_config.yml
usage: seal [-h] [-c CONFIG] [-v] (-i INVENTORY_FILE | --inventory-kubernetes)
[--open-stack-cloud OPEN_STACK_CLOUD] [--remote-user REMOTE_USER]
[--kube-config KUBE_CONFIG]
(--validate-policy-file VALIDATE_POLICY_FILE | --run-policy-file RUN_POLICY_FILE | --interactive)
seal: error: one of the arguments -i/--inventory-file --inventory-kubernetes is required
A lot of people approached me to ask about the python 2.7 support.
If you need it for your setup, please vote for this issue.
flake8 testing of https://github.com/bloomberg/powerfulseal on Python 3.6.3
$ flake8 . --count --select=E901,E999,F821,F822,F823 --show-source --statistics
./powerfulseal/cli/pscmd.py:334:16: F821 undefined name 'op'
if op == "namespace":
^
https://travis-ci.org/bloomberg/powerfulseal/jobs/319560719#L612-L616
Our application supports running on VMware and Openstack as platform, We need guidance on how we can add Vmware as cloud option.
We need to run chaos test against VMware as platform
The PolicyRunner.run
method supports a maximum number of times to run (loops=
named parameter) but this is not exposed as far as I can see in a useful manner.
A CLI flag of the maximum number of loops would be useful for testers/scripters who'd like to run PS scenarios a set number of times and then exit, to present test run results.
(I have this exposure implemented privately, and am willing to contribute it, but wanted to get the idea vetted first)
Scenario: powerfulseal is hosted in a cluster it is chaosing. It would be useful to be able negative match and filter logic - that is the ability to say "match nodes that are NOT ip-foo-bar" and "filter for nodes that are NOT ip-foo-bar" to allow powerfulseal to do safer node kills in this situation
Because property_match uses a regex SOME of this functionality could, with some work, be done. The addition of a property_not mapping, however, would provide easier negation logic for policy writers
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.