Comments (72)
You can do exec
over websockets instead of SPDY. So would be good to investigate websockets instead of the sdpy route (which will be osbolete anyway).
from python.
@mbohlool @mrmcmuffinz : i have fixed the connect_post_namespaced_pod_exec in #123 - added a e2e test as well to show that it works fine and we don't regress when we generate new client from swagger.
from python.
assert_hostname is not in config package, I understand the confusion though, we should clear that up.
on a side note, you don't need to pass kube config location if it is in the default location (as I see it is for your case).
from python.
Well I don't mind creating a new issue for post work. The access to stderr and stdout separately was one of my comments on original PR too. The majority part of the task is done so I will leave it to you and @dims if you want to close this and create a new issue, but I don't think our work on exec calls is done until we solve those issues. I would like to have a similar approach to Watch here. Will try something today.
from python.
from python.
realted: http://stackoverflow.com/questions/37349440/upgrade-request-required-when-running-exec-in-kubernetes
I guess if we change code generator to accept headers, it should be possible to pass upgrade headers to the call.
Update: Look like we need an SPDY client for python too.
from python.
Is it feasible to have it soon? It would be nice to have a date in order to know if I can use it within my deadline. If not, do you have some workaround in mind? Thank you in advance.
from python.
I don't have a date yet, the problem is urllib3 does not support SPDY protocol. we need to use something like python-spdylay. Unless somebody own this (or I found the time to look at it in more depth) I can't put a date on it.
from python.
Just spend a little more time with python-spdylay. Even their client example is not working and it does not seem to be actively developed/supported. If anybody have any idea of how to talk SPDY in python, please let me know.
from python.
More information here: kubernetes/kubernetes#24668
from python.
Moreover, there is kubernetes/kubernetes#7452 which stands that SPDY is deprecated and HTTP/2 might be the right option, but looks like it is still hanging in the air. Anyways, it is SPDY what is working for current versions so we need to somehow find a workaround.
from python.
A little more digging pointed me to a library called thor. I will give it a try. If you can try it too to see if you can establish a SPDY connection with it, that would be nice. My assumption is SPDY is always secure so there should be a way to specify client-cert and other cert files that we normally set in kubernetes.client.configuration
class.
from python.
Thanks @sebgoa it look like websockets are supported as a fall-back: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api-conventions.md#websockets-and-spdy
from python.
@mbohlool is there an example that you could provide demonstrating the use of exec via websockets?
from python.
@mrmcmuffinz I did not have time to do it. That is why I added "Help needed" to this issue so somebody from the community hopefully help.
from python.
@mbohlool My apologies, I thought you had some example. Do you happen to know who we can talk to, to help out? Can we work out defining what exactly it takes to get this done?
from python.
Adding my stackoverflow question here too, for other people who stumble across this. Also has anyone looked into speaking with the developers of urilib3
to see if they will support http/2
? That way there is a minimum impact to the api_client...
from python.
The ApiException for exec or any call needs SPDY/Websockets is informative. The right way to fix this is to catch that exception in api_client.py and start a websocket call instead. For the first step, try to write a python code that does not use this repo, and directly call into exec REST API using websocket. use kubectl proxy to get rid of authorization part and call into local host for simplification.
from python.
I actually tried doing that but I could not figure out how to setup the authentication part of the code for the websocket. I have three pem certificates that were given to me and I'm just not sure how that fits into the picture. Thank you for your response btw.
from python.
have you try kubectl proxy? that way you should not worry about authentication for now. it will proxy your call from localhost with the right authentication headers to the remote host.
from python.
Ohh wow I totally forgot about the kubectl proxy command, if I do that should I switch my url to localhost in the websocket url versus the actual endpoint itself?
Edit:
nvm, I reread again and that is indeed what you suggested I do.
from python.
Yes. lets have a working code with proxy then think about how to incorporate it into the client-python.
from python.
Yeah I think I'm a bit further however I get a connection reset by peer error right now. I think I will need to read up on sockets in python as well as what all I need to do in order to setup the appropriate message for communicating. FYI: I'm using a package called hyper for the actual communication unless you can suggest anything better.
from python.
Here is the issue against urllib3 for http/2
suuport urllib3/urllib3#836.
from python.
@mrmcmuffinz I was wondering how you are testing this. I was doing some basic testing with hyper; but the exec call needs a "SPDY/3.1" upgrade header that hyper doesn't know how to handle.
If you do a connection upgrade with "SPDY/3.1" you will get this exception: https://github.com/Lukasa/hyper/blob/24b3b37d226bedb6199f22b788e066b91dbf892a/hyper/common/conncetion.py#L130
And if you do an h2 upgrade you will get this from kubernetes (since http2 is not implemented yet):
https://github.com/kubernetes/kubernetes/blob/55bee3ad21f025b1416a4e1f10de753f484b66d3/pkg/util/httpstream/spdy/upgrade.go#L48
from python.
Hi @chekolyn in all honesty I put it on hold and started looking at other clients. I just wanted to use these python client with py.test for FVT but that kinda went out the door because of this limitation.
Edit:
As for the socket workaround that was another rabbit hole that I did not want to go into right now.
from python.
OK guys a gave websockets a try and I have a very limited POC
I can confirm websockets works!
(Tested on OpenShift)
# POC for Websocket exec calls to pods:
#
# Channel 0 is STDIN, 1 is STDOUT, 2 is STDERR (if TTY is not requested),
# and 3 is a special error channel. The server only reads from STDIN,
# writes to the other three.
import websocket
import ssl
from subprocess import check_output
def on_message(ws, message):
print 'message received ..'
print message
def on_error(ws, error):
print 'error happened .. '
print error
def on_close(ws):
print "### closed ###"
def on_open(ws):
print 'Opening Websocket connection to the server ... '
# This session_key I got, need to be passed over websocket header isntad
# of ws.send.
#ws.send(session_key)
token = check_output(["oc", "whoami", "-t"])
pod = "router-1-8u540"
exec_command = "uptime".replace(" ", "+")
url = "wss://localhost:8443/api/v1/namespaces/default/pods/%s/exec?command=%s&stderr=true&stdin=true&stdout=true&tty=false" % (pod, exec_command)
header = "Authorization: Bearer " + token
if __name__ == "__main__":
websocket.enableTrace(False)
ws = websocket.WebSocketApp(url,
on_message = on_message,
on_error = on_error,
on_close = on_close,
header = [header]
)
ws.on_open = on_open
ws.run_forever(sslopt={"cert_reqs": ssl.CERT_NONE})
# ws.on_message = on_message
# ws.run_forever(sslopt={"cert_reqs": ssl.CERT_NONE})
from python.
@chekolyn "oc" implies openshift? does this work in a pure kubernetes environment?
from python.
@dims I haven't tested it on a pure k8s but I believe It should work, the oc command is just grabbing the current auth token to authenticate via headers (perhaps this is the only portion that it's OpenShift specific in the POC, you can probably remove the check_output for a valid token and it should work too ). The url api exec call is pure kubernetes and websockets calls seem to be working fine.
I'm basically using this authentication strategy in the api call: https://kubernetes.io/docs/user-guide/accessing-the-cluster/#without-kubectl-proxy-post-v13x
from python.
Hi, has anyone else validated that these changes have indeed solved our exec demands? Do I need to change my code which is included in the stackoverflow link above or should I be good to go?
from python.
@mrmcmuffinz did you build the client from master? The change just merged into master and will be available in the next release, but you should be able to follow the guide in README file to install client from master and give it a try. Let us know if that worked.
from python.
Hi @mbohlool I have not had the chance yet however I found it odd that there was some deliveries but no review comments when I looked at the commits. Also I read that there were no tests delivered as part of the delivery and that the auto generator would potentially overwrite some of the code... I will certainly be taking a look at this over the weekend. Thanks to those who were involved!
from python.
@mrmcmuffinz : we have an e2e test that actually runs k8s and tests the exec code path - https://github.com/kubernetes-incubator/client-python/blob/master/kubernetes/e2e_test/test_client.py#L75
from python.
@mrmcmuffinz look at #120 for more accurate information. There are lots of comments and e2e tests. Also we excluded those files from auto-generation. Let me know how was your test.
from python.
Yeah sorry for the misleading information, I should have dug a bit deeper, saw this dims@7e9adec, into the commits versus just reading some of the commits I saw as part of the commits above.
from python.
I cloned the repo and tried installing the client but it still does not work for me...
[acabrer@mrmcmuffins client-python]$ python2.7 setup.py build
/usr/local/lib/python2.7/site-packages/setuptools/dist.py:340: UserWarning: The version specified ('1.0.0-snapshot') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
"details." % self.metadata.version
running build
running build_py
running egg_info
writing requirements to kubernetes.egg-info/requires.txt
writing kubernetes.egg-info/PKG-INFO
writing top-level names to kubernetes.egg-info/top_level.txt
writing dependency_links to kubernetes.egg-info/dependency_links.txt
reading manifest file 'kubernetes.egg-info/SOURCES.txt'
writing manifest file 'kubernetes.egg-info/SOURCES.txt'
[91541 refs]
Seems this part was not clear enough:
[acabrer@mrmcmuffins client-python]$ sudo python2.7 setup.py develop
[sudo] password for acabrer:
/usr/lib/python2.7/site-packages/setuptools/dist.py:294: UserWarning: The version specified ('1.0.0-snapshot') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
"details." % self.metadata.version
running develop
running egg_info
writing requirements to kubernetes.egg-info/requires.txt
writing kubernetes.egg-info/PKG-INFO
writing top-level names to kubernetes.egg-info/top_level.txt
writing dependency_links to kubernetes.egg-info/dependency_links.txt
reading manifest file 'kubernetes.egg-info/SOURCES.txt'
writing manifest file 'kubernetes.egg-info/SOURCES.txt'
running build_ext
Creating /usr/lib/python2.7/site-packages/kubernetes.egg-link (link to .)
kubernetes 1.0.0-snapshot is already the active version in easy-install.pth
Installed /home/acabrer/git.workspace/client-python
Processing dependencies for kubernetes===1.0.0-snapshot
Searching for ipaddress==1.0.18
Best match: ipaddress 1.0.18
Adding ipaddress 1.0.18 to easy-install.pth file
Using /usr/lib/python2.7/site-packages
Searching for oauth2client==4.0.0
Best match: oauth2client 4.0.0
Adding oauth2client 4.0.0 to easy-install.pth file
Using /usr/lib/python2.7/site-packages
Searching for PyYAML==3.12
Best match: PyYAML 3.12
Adding PyYAML 3.12 to easy-install.pth file
Using /usr/lib64/python2.7/site-packages
Searching for python-dateutil==2.6.0
Best match: python-dateutil 2.6.0
Processing python_dateutil-2.6.0-py2.7.egg
python-dateutil 2.6.0 is already the active version in easy-install.pth
Using /usr/lib/python2.7/site-packages/python_dateutil-2.6.0-py2.7.egg
Searching for certifi==2016.9.26
Best match: certifi 2016.9.26
Processing certifi-2016.9.26-py2.7.egg
certifi 2016.9.26 is already the active version in easy-install.pth
Using /usr/lib/python2.7/site-packages/certifi-2016.9.26-py2.7.egg
Searching for six==1.10.0
Best match: six 1.10.0
Adding six 1.10.0 to easy-install.pth file
Using /usr/lib/python2.7/site-packages
Searching for urllib3==1.20
Best match: urllib3 1.20
Adding urllib3 1.20 to easy-install.pth file
Using /usr/lib/python2.7/site-packages
Searching for httplib2==0.9.2
Best match: httplib2 0.9.2
Adding httplib2 0.9.2 to easy-install.pth file
Using /usr/lib/python2.7/site-packages
Searching for pyasn1==0.1.9
Best match: pyasn1 0.1.9
Processing pyasn1-0.1.9-py2.7.egg
pyasn1 0.1.9 is already the active version in easy-install.pth
Using /usr/lib/python2.7/site-packages/pyasn1-0.1.9-py2.7.egg
Searching for rsa==3.4.2
Best match: rsa 3.4.2
Adding rsa 3.4.2 to easy-install.pth file
Installing pyrsa-encrypt-bigfile script to /usr/bin
Installing pyrsa-encrypt script to /usr/bin
Installing pyrsa-verify script to /usr/bin
Installing pyrsa-sign script to /usr/bin
Installing pyrsa-priv2pub script to /usr/bin
Installing pyrsa-decrypt script to /usr/bin
Installing pyrsa-decrypt-bigfile script to /usr/bin
Installing pyrsa-keygen script to /usr/bin
Using /usr/lib/python2.7/site-packages
Searching for pyasn1-modules==0.0.8
Best match: pyasn1-modules 0.0.8
Adding pyasn1-modules 0.0.8 to easy-install.pth file
Using /usr/lib/python2.7/site-packages
Finished processing dependencies for kubernetes===1.0.0-snapshot
>>> response = cli.connect_post_namespaced_pod_exec(pod, ns, stderr=True, stdin=True, stdout=True, command=exec_command)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "kubernetes/client/apis/core_v1_api.py", line 3396, in connect_post_namespaced_pod_exec
(data) = self.connect_post_namespaced_pod_exec_with_http_info(name, namespace, **kwargs)
File "kubernetes/client/apis/core_v1_api.py", line 3501, in connect_post_namespaced_pod_exec_with_http_info
collection_formats=collection_formats)
File "kubernetes/client/api_client.py", line 329, in call_api
_return_http_data_only, collection_formats, _preload_content, _request_timeout)
File "kubernetes/client/api_client.py", line 153, in __call_api
_request_timeout=_request_timeout)
File "kubernetes/client/api_client.py", line 383, in request
body=body)
File "kubernetes/client/rest.py", line 275, in POST
body=body)
File "kubernetes/client/rest.py", line 231, in request
raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 09 Feb 2017 04:23:14 GMT', 'Content-Length': '139', 'Content-Type': 'application/json'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Upgrade request required","reason":"BadRequest","code":400}
Am I missing something?
[acabrer@mrmcmuffins client-python]$ python2.7 -m pip list --format=columns |grep kubernetes
kubernetes 1.0.0-snapshot
from python.
you need to install it not build.
$ python2.7 setup.py install
I suggest you create a clean virtualenv first to make sure you are not conflicting with any other installed package.
from python.
I did a develop... but none the less I will just uninstall and reinstall
from python.
@mrmcmuffinz : If you see the test i pointed out to, you will see connect_get_namespaced_pod_exec works. to get connect_post_namespaced_pod_exec to work, please patch the following line to allow "POST"
https://github.com/kubernetes-incubator/client-python/blob/master/kubernetes/client/api_client.py#L349
from python.
Yeah I saw the test later, sorry about that however for your patch suggestion that does not sound like a good idea to me. Perhaps it will get to work in my case however that breaks assumptions such that you forwarded the post request via get...
from python.
There is a requirements.txt which I was not aware of until I looked into the commit comments... after installing the requirements file
(kpclient)[acabrer@mrmcmuffins bin]$ python -m pip install -r ~/git.workspace/client-python/requirements.txt
Requirement already satisfied (use --upgrade to upgrade): certifi>=14.05.14 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/certifi-2017.1.23-py2.7.egg (from -r /home/acabrer/git.workspace/client-python/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): six>=1.9.0 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/six-1.10.0-py2.7.egg (from -r /home/acabrer/git.workspace/client-python/requirements.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): python-dateutil>=2.5.3 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/python_dateutil-2.6.0-py2.7.egg (from -r /home/acabrer/git.workspace/client-python/requirements.txt (line 3))
Collecting setuptools>=21.0.0 (from -r /home/acabrer/git.workspace/client-python/requirements.txt (line 4))
Downloading setuptools-34.1.1-py2.py3-none-any.whl (389kB)
100% |████████████████████████████████| 399kB 1.9MB/s
Requirement already satisfied (use --upgrade to upgrade): urllib3>=1.19.1 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/urllib3-1.20-py2.7.egg (from -r /home/acabrer/git.workspace/client-python/requirements.txt (line 5))
Requirement already satisfied (use --upgrade to upgrade): pyyaml>=3.12 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/PyYAML-3.12-py2.7-linux-x86_64.egg (from -r /home/acabrer/git.workspace/client-python/requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): oauth2client>=4.0.0 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/oauth2client-4.0.0-py2.7.egg (from -r /home/acabrer/git.workspace/client-python/requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): ipaddress>=1.0.17 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/ipaddress-1.0.18-py2.7.egg (from -r /home/acabrer/git.workspace/client-python/requirements.txt (line 8))
Collecting websocket-client>=0.32.0 (from -r /home/acabrer/git.workspace/client-python/requirements.txt (line 9))
Downloading websocket_client-0.40.0.tar.gz (196kB)
100% |████████████████████████████████| 204kB 2.5MB/s
Collecting packaging>=16.8 (from setuptools>=21.0.0->-r /home/acabrer/git.workspace/client-python/requirements.txt (line 4))
Downloading packaging-16.8-py2.py3-none-any.whl
Collecting appdirs>=1.4.0 (from setuptools>=21.0.0->-r /home/acabrer/git.workspace/client-python/requirements.txt (line 4))
Downloading appdirs-1.4.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): httplib2>=0.9.1 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/httplib2-0.10.3-py2.7.egg (from oauth2client>=4.0.0->-r /home/acabrer/git.workspace/client-python/requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): pyasn1>=0.1.7 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/pyasn1-0.2.2-py2.7.egg (from oauth2client>=4.0.0->-r /home/acabrer/git.workspace/client-python/requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): pyasn1-modules>=0.0.5 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/pyasn1_modules-0.0.8-py2.7.egg (from oauth2client>=4.0.0->-r /home/acabrer/git.workspace/client-python/requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): rsa>=3.1.4 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/rsa-3.4.2-py2.7.egg (from oauth2client>=4.0.0->-r /home/acabrer/git.workspace/client-python/requirements.txt (line 7))
Collecting backports.ssl-match-hostname (from websocket-client>=0.32.0->-r /home/acabrer/git.workspace/client-python/requirements.txt (line 9))
Collecting pyparsing (from packaging>=16.8->setuptools>=21.0.0->-r /home/acabrer/git.workspace/client-python/requirements.txt (line 4))
Downloading pyparsing-2.1.10-py2.py3-none-any.whl (56kB)
100% |████████████████████████████████| 61kB 4.5MB/s
Installing collected packages: pyparsing, packaging, appdirs, setuptools, backports.ssl-match-hostname, websocket-client
Found existing installation: setuptools 20.3
Uninstalling setuptools-20.3:
Successfully uninstalled setuptools-20.3
Running setup.py install for websocket-client ... done
Successfully installed appdirs-1.4.0 backports.ssl-match-hostname-3.5.0.1 packaging-16.8 pyparsing-2.1.10 setuptools-34.1.1 websocket-client-0.40.0
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
I get the below:
(kpclient)[acabrer@mrmcmuffins bin]$ ~/kube.py
[kube.py:25 - main() ] (0)
Reason: hostname 'x.x.x.x' doesn't match u'10.10.10.1'
I'm guessing this is because I need to do the proxy command in kubectl....
from python.
Hi @dims thanks for the update, I have pulled down the latest code and installed it however I still have the below error
(kpclient)[acabrer@mrmcmuffins bin]$ ~/kube.py
[kube.py:25 - main() ] (0)
Reason: hostname 'x.x.x.x' doesn't match u'10.10.10.1'
Traceback (most recent call last):
File "/home/acabrer/kube.py", line 22, in main
response = cli.connect_post_namespaced_pod_exec(pod, ns, stderr=True, stdin=True, stdout=True, command=exec_command, tty=False)
File "build/bdist.linux-x86_64/egg/kubernetes/client/apis/core_v1_api.py", line 3396, in connect_post_namespaced_pod_exec
(data) = self.connect_post_namespaced_pod_exec_with_http_info(name, namespace, **kwargs)
File "build/bdist.linux-x86_64/egg/kubernetes/client/apis/core_v1_api.py", line 3501, in connect_post_namespaced_pod_exec_with_http_info
collection_formats=collection_formats)
File "build/bdist.linux-x86_64/egg/kubernetes/client/api_client.py", line 329, in call_api
_return_http_data_only, collection_formats, _preload_content, _request_timeout)
File "build/bdist.linux-x86_64/egg/kubernetes/client/api_client.py", line 153, in __call_api
_request_timeout=_request_timeout)
File "build/bdist.linux-x86_64/egg/kubernetes/client/api_client.py", line 354, in request
headers=headers)
File "build/bdist.linux-x86_64/egg/kubernetes/client/ws_client.py", line 112, in GET
reason='\n'.join([str(error) for error in client.errors])
ApiException: (0)
from python.
try setting configuration.assert_hostname
to False
.
from python.
Do you mean kubernetes.config
?
from python.
@mbohlool just tried that as well still does not work.
from python.
@mbohlool
Ok here is what I did, I added the line you suggested to https://github.com/kubernetes-incubator/client-python/blob/master/kubernetes/client/ws_client.py#L107 and it worked for me. However I had set that to my config object and it was not taking affect.
I'm just running a pwd and it looks like it worked for me:
(kpclient)[acabrer@mrmcmuffins client-python]$ ~/kube.py
'/\n'
Just to add and confirm a bit more, I can
exec_command = ['/bin/sh', '-c', 'cat /etc/centos-release']
on one of my pods.
(kpclient)[acabrer@mrmcmuffins client-python]$ ~/kube.py
CentOS Linux release 7.3.1611 (Core)
from python.
it should be something like:
from kubernetes.client import configuration
from kubernetes.config import load_kube_config
load_kube_config()
configuration.assert_hostname = False
Set the flag after you loaded the config.
from python.
I was doing:
from kubernetes.client.rest import ApiException
path_to_config = "/home/acabrer/.kube/config"
config.load_kube_config(config_file=path_to_config)
config.assert_hostname = False
cli = client.CoreV1Api()
I can certainly try your suggestion.
from python.
Yeah, I'm going to be writing some py.test fixtures to pass in the location of the configuration hence why I was using the above code. In the code you are suggestion how would one load a config from non-default location, does the load_kube_config()
have a optional argument to pass it in?
from python.
To answer my own question yes it does https://github.com/kubernetes-incubator/client-python/blob/master/kubernetes/config/kube_config.py#L283
from python.
@mbohlool - we should close this one out.
from python.
I still think the way it's implemented is a little hacky (not the implementation itself but the way we integrated it into the api call by checking an specific API endpoint). I would like to keep this issue a little longer so we can look back and clean it up a little. I will take another look tomorrow.
from python.
Hi might I bring my two cents. I still think we need to fix the ged and post requests for exec as they both go down the same path. Lastly I have created a dozen or so tests for the product I'm working on but there is one thing that I don't understand, where does one access the stderr, stdout? All we get back is a response message but those elements are not accessible in any specific manner. I think the api is a bit misleading though perhaps I'm misinterpreting it. However that being said this Git issue was opened to add exec
support via websockets and @dims did an excellent job at doing that, despite me reporting some post work needed. Which leads me to believe that what you(@mbohol) are asking for is beyond the extent of the original scope for this item.
from python.
I'm ok with either approach @mbohlool and will provide my feedback along the way as now I'm kinda a "stakeholder" in this. Any idea when we will have a new release with these changes, or at least a git tag for marking this particular milestone?
from python.
I have a WIP PR that will send out soon. Only thing that is not working is stdin. stdout and stderr is working as well as streaming output. I will do another beta release after that PR.
from python.
Ok, please keep me posted.
from python.
@mrmcmuffinz I sent a partial PR. stdin is not working yet (and I am not sure why). You also show concern about get/post going the same path, what is your concern? as far as I can tell websocket does not have a method like http. so get/post is the same for websocket.
from python.
I sent a partial PR. stdin is not working yet
Do you need some review on this or are you fine?
What is your concern for get/post going down the same path?
Yeah I realized later and came to the same conclusion, which was that the websocket protocol does not deal with get and post requests however the intent I believe was to have websockets as a failback mechanism to the rest api and what we have right now is it just default for exec
for get/post. I think we should have that separated out such that if it is get, it goes down the get path and vice versa for post. That makes it easier to fix when we the refactor comes into play. At least that is my two cents on that point.
Edit 1:
I looked at some of the code you wrote and have a few questions:
- I'd like to suggest first if we can agree on below before we come up with some implementation?
File descriptor stderr
standard error or stream 2
File descriptor stdout
standard out or stream 1
File descriptor stdin
standard input or input stream.
-
Can we agree that the way the above
std
streams get externalize, they should be externalized via one way for both websockets and rest? -
As for
stdout
andstderr
I would like to be able to access those in the response body via some member, example below, do you think this right?
response.stdout
response.stderr
-
Does it really make sense to exertnalize
stdin
right now and if it does what does that mean forexec
calls? -
Lastly, how do I access the return code of the
exec
command I just executed?
from python.
The normal HTTP call (that I assume you mean that by rest
) for exec does not work and will fail for exec call.writing to stdin would allow interaction with pod. It is much like kubctl exec command when you attach to stdin.
from python.
@mbohlool I meet the following error when trying to execute a command in the pod.
Exception when calling connect_post_namespaced_pod_exec: (0)
Reason: 'NoneType' object is not iterable
I tried both k8s 1.2 and 1.5 cluster. It seems the websockets is not accessable so it fails when calling WebSocket.connect() method. How to enable websockets on the k8s master node?
from python.
I don't think the issue is the target cluster (I don't think websocket can be disabled, it is always enabled). Would you share your code (fragment)? This can be a bug in our implementation.
from python.
@mbohlool I actually edited my prior reply and not sure if you saw the full message could you kindly please revisit it?
@zhenhua Could you kindly please include apart from the fragment/redacted source code but also the stack trace?
from python.
- That is what I have in PR, right?
- what do you mean by
rest
? you mean normal http call? we don't support that. if you mean a blocking call then this question is the same as 3? - Good idea.
- kubectl let you connect to a container using exec (I think the option is
-stdin
that Passes stdin to the container/pod). This let you interact with container like the last part of my example (that is not working right now and I disabled it) - By return code you mean return code of the command you ran on the container? You should somehow output it and access it from the output (something like
echo $?
).
from python.
My test code is straight forward enough. Here it is
def _exec_pod_cmd(cli, pod_name, command):
try:
resp = cli.connect_post_namespaced_pod_exec(
name=pod_name, namespace=namespace, command=command)
return resp
except ApiException as e:
print("Exception when calling connect_post_namespaced_pod_exec: %s\n" % e)
config.load_kube_config(config_file=path_to_config)
config.assert_hostname = False
cli = client.CoreV1Api()
resp = _list_namespaced_pod(cli, str("name=%s" % rc_name))
pod_name = resp.items[0].metadata.name
print("pod name %s " % pod_name)
resp = _exec_pod_cmd(cli, pod_name, "pwd")
pprint(resp)
Here is some debug trace:
(Pdb) b 108
Breakpoint 5 at /usr/lib/python2.7/site-packages/kubernetes/client/ws_client.py:108
(Pdb) c
> /usr/lib/python2.7/site-packages/kubernetes/client/ws_client.py(108)GET()
-> client = WSClient(configuration, url, headers)
(Pdb) p url
'ws://10.13.10.63:8080/api/v1/namespaces/289-ns/pods/test-rc-jd2rp/exec&command=pwd'
(Pdb) p configuration
<kubernetes.client.configuration.ConfigurationObject object at 0x3012290>
(Pdb) p headers
{'Content-Type': 'application/json', 'Accept': '*/*', 'User-Agent': 'Swagger-Codegen/1.0.0-snapshot/python'}
(Pdb)
After invoking websocket
(Pdb) b 181
Breakpoint 7 at /usr/lib/python2.7/site-packages/websocket/_app.py:181
(Pdb) c
> /usr/lib/python2.7/site-packages/websocket/_app.py(181)run_forever()
-> host=host, origin=origin)
(Pdb) l
176 self.sock.connect(self.url, header=self.header, cookie=self.cookie,
177 http_proxy_host=http_proxy_host,
178 http_proxy_port=http_proxy_port,
179 http_no_proxy=http_no_proxy, http_proxy_auth=http_proxy_auth,
180 subprotocols=self.subprotocols,
181 B-> host=host, origin=origin)
182 self._callback(self.on_open)
183
184 if ping_interval:
185 event = threading.Event()
186 thread = threading.Thread(target=self._send_ping, args=(ping_interval, event))
(Pdb) p self.url
'ws://10.13.10.63:8080/api/v1/namespaces/289-ns/pods/test-rc-jd2rp/exec&command=pwd'
(Pdb) p http_proxy_host
None
(Pdb) p self.header
None
(Pdb) n
TypeError: TypeErro...erable",)
> /usr/lib/python2.7/site-packages/websocket/_app.py(181)run_forever()
-> host=host, origin=origin)
(Pdb) n
> /usr/lib/python2.7/site-packages/websocket/_app.py(219)run_forever()
-> except (Exception, KeyboardInterrupt, SystemExit) as e:
(Pdb) p Exception
<type 'exceptions.Exception'>
(Pdb) n
> /usr/lib/python2.7/site-packages/websocket/_app.py(220)run_forever()
-> self._callback(self.on_error, e)
(Pdb) p e
TypeError("'NoneType' object is not iterable",)
(Pdb)
Let me ask a stupid question: How to access the websocket, port 8443 or port 8080?
from python.
@mrmcmuffinz: for no 3. It is a good idea, however API spec defined the return type as string. If you want to get stdout and stderr separately you need to call the API with _preload_content=False
and read from each channel separately.
resp = api.connect_get_namespaced_pod_exec(name, 'default',
command=exec_command,
stderr=True, stdin=True,
stdout=True, tty=False,
_preload_content=False)
resp.run_forever()
if resp.peek_stdout(): print("STDOUT: ", resp.read_stdout())
if resp.peek_stderr(): print("STDERR: ", resp.read_stderr())
from python.
@zhenhua, I guess your problem is header
. it should not be None
. It is fixed as part of my PR.
from python.
@mbohlool, thanks. I use your mbohlool:exec branch to verify it on my side and it works fine. Only one thing is that, the container parameter of connect_post_namespaced_pod_exec() call is NOT optional, otherwise, it will return an 404 error of 'HTTP/1.1 404 Not Found'.
from python.
@zhenhua documentation of the parameter says: "Defaults to only container if there is only one container in the pod." I think in case of our examples and e2e tests we have one container but you may have more than one container in your pod. Happy that it worked, I will merge the PR and cut another beta release.
from python.
@mbohlool You are right. Actually, when creating a pod in my test, there're always have at least two containers, one is registry.access.redhat.com/rhel7/pod-infrastructure:latest and another is my test container. So I have to always specify the container name.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
63aa2b961a10 registry.v2.service.xxx.org/dongyue/xxx-base-centos7:20170111 "/bin/sh -c /usr/sbin" 40 minutes ago Up 40 minutes k8s_test-rc.12ee117e_test-rc-t4052_289-ns_0b32083c-f806-11e6-b075-c81f66cd4467_1c38b8b6
be4cc79af38c registry.access.redhat.com/rhel7/pod-infrastructure:latest "/pod" 40 minutes ago Up 40 minutes k8s_POD.ae8ee9ac_test-rc-t4052_289-ns_0b32083c-f806-11e6-b075-c81f66cd4467_ffa70144
from python.
hi,
i test with websocket, but got error message "x509: certificate is valid for 127.0.0.1, not xx.xx.xx.xx"
i use golang like :
wsurl := "wss://xx.xx.xx.xx:8080/r/projects/1a92/kubernetes:6443/api/v1/namespaces/NM/pods/testPod-546cdd8d79-7h8nv/exec?command=ls&container=testPod&stderr=true&stdin=true&stdout=true&tty=false"
u, err := neturl.Parse(wsurl)
rawConn, err := net.Dial("tcp", u.Host)
wsHeaders := http.Header{
"Authorization": {"Bearer "+env_bearer_token},
"Origin": {"https://xx.xx.xx.xx:8080/r/projects/1a92/kubernetes:6443"},
"Sec-WebSocket-Extensions": {"permessage-deflate; client_max_window_bits, x-webkit-deflate-frame"},
}
wsConn, resp, err := websocket.NewClient(rawConn, u, wsHeaders, 1024, 1024)
anything wrong ? thanks~
from python.
Hi! I'm getting this same error now in version 8.0.1
Here's the offending code:
v1 = client.CoreV1Api()
all_pods = [pod.metadata.name for pod in v1.list_namespaced_pod(self.namespace).items]
pod = next(pod for pod in all_pods if self.name in pod)
api_response = v1.connect_get_namespaced_pod_exec(
pod, self.namespace,
container=self.pod_template.containers[0].name,
command=['ps', 'axu'],
stdin=False, stdout=True, stderr=True,
tty=False
)
And here's the exception that is being raised:
self = <kubernetes.client.rest.RESTClientObject object at 0x7f870f136cc0>, method = 'GET', url = 'https://****************:443/api/v1/namespaces/elastictests-ozxuefet/pods/test-elastic-cluster-77cf4bfdf9-rsslx/exec'
query_params = [('command', ['ps', 'axu']), ('container', 'elastic-0'), ('stderr', True), ('stdin', False), ('stdout', True), ('tty', False)]
headers = {'Accept': '*/*', 'Content-Type': 'application/json', 'User-Agent': 'Swagger-Codegen/8.0.1/python', 'authorization': 'Bearer f725ca479019a546e359f8a5b7ce9b9d'}, body = None, post_params = {}, _preload_content = True, _request_timeout = None
def request(self, method, url, query_params=None, headers=None,
body=None, post_params=None, _preload_content=True, _request_timeout=None):
"""
:param method: http request method
:param url: http request url
:param query_params: query parameters in the url
:param headers: http request headers
:param body: request json body, for `application/json`
:param post_params: request post parameters,
`application/x-www-form-urlencoded`
and `multipart/form-data`
:param _preload_content: if False, the urllib3.HTTPResponse object will be returned without
reading/decoding response data. Default is True.
:param _request_timeout: timeout setting for this request. If one number provided, it will be total request
timeout. It can also be a pair (tuple) of (connection, read) timeouts.
"""
method = method.upper()
assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', 'PATCH', 'OPTIONS']
if post_params and body:
raise ValueError(
"body parameter cannot be used with post_params parameter."
)
post_params = post_params or {}
headers = headers or {}
timeout = None
if _request_timeout:
if isinstance(_request_timeout, (int, ) if PY3 else (int, long)):
timeout = urllib3.Timeout(total=_request_timeout)
elif isinstance(_request_timeout, tuple) and len(_request_timeout) == 2:
timeout = urllib3.Timeout(connect=_request_timeout[0], read=_request_timeout[1])
if 'Content-Type' not in headers:
headers['Content-Type'] = 'application/json'
try:
# For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
if query_params:
url += '?' + urlencode(query_params)
if re.search('json', headers['Content-Type'], re.IGNORECASE):
if headers['Content-Type'] == 'application/json-patch+json':
if not isinstance(body, list):
headers['Content-Type'] = \
'application/strategic-merge-patch+json'
request_body = None
if body is not None:
request_body = json.dumps(body)
r = self.pool_manager.request(method, url,
body=request_body,
preload_content=_preload_content,
timeout=timeout,
headers=headers)
elif headers['Content-Type'] == 'application/x-www-form-urlencoded':
r = self.pool_manager.request(method, url,
fields=post_params,
encode_multipart=False,
preload_content=_preload_content,
timeout=timeout,
headers=headers)
elif headers['Content-Type'] == 'multipart/form-data':
# must del headers['Content-Type'], or the correct Content-Type
# which generated by urllib3 will be overwritten.
del headers['Content-Type']
r = self.pool_manager.request(method, url,
fields=post_params,
encode_multipart=True,
preload_content=_preload_content,
timeout=timeout,
headers=headers)
# Pass a `string` parameter directly in the body to support
# other content types than Json when `body` argument is provided
# in serialized form
elif isinstance(body, str):
request_body = body
r = self.pool_manager.request(method, url,
body=request_body,
preload_content=_preload_content,
timeout=timeout,
headers=headers)
else:
# Cannot generate the request from given parameters
msg = """Cannot prepare a request message for provided arguments.
Please check that your arguments match declared content type."""
raise ApiException(status=0, reason=msg)
# For `GET`, `HEAD`
else:
r = self.pool_manager.request(method, url,
fields=query_params,
preload_content=_preload_content,
timeout=timeout,
headers=headers)
except urllib3.exceptions.SSLError as e:
msg = "{0}\n{1}".format(type(e).__name__, str(e))
raise ApiException(status=0, reason=msg)
if _preload_content:
r = RESTResponse(r)
# In the python 3, the response.data is bytes.
# we need to decode it to string.
if PY3:
r.data = r.data.decode('utf8')
# log response body
logger.debug("response body: %s", r.data)
if not 200 <= r.status <= 299:
> raise ApiException(http_resp=r)
E kubernetes.client.rest.ApiException: (400)
E Reason: Bad Request
E HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Thu, 07 Feb 2019 14:03:23 GMT', 'Content-Length': '139'})
E HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Upgrade request required","reason":"BadRequest","code":400}
/root/.local/share/virtualenvs/stratio-atf-4Tz-qxfE/lib/python3.7/site-packages/kubernetes/client/rest.py:222: ApiException
I'm using pipenv to manage the working environment. Here are the packages I have installed and corresponding versions:
In [11]: vs = json.loads(open('Pipfile.lock','r').read())
In [12]: [(i, vs['default'][i]['version']) for i in vs['default']]
Out[12]:
[('adal', '==1.2.1'),
('apipkg', '==1.5'),
('asn1crypto', '==0.24.0'),
('atomicwrites', '==1.3.0'),
('attrs', '==18.2.0'),
('cachetools', '==3.1.0'),
('certifi', '==2018.11.29'),
('cffi', '==1.11.5'),
('chardet', '==3.0.4'),
('coverage', '==4.5.2'),
('cryptography', '==2.5'),
('elasticsearch', '==6.3.1'),
('execnet', '==1.5.0'),
('google-auth', '==1.6.2'),
('idna', '==2.8'),
('kubernetes', '==8.0.1'),
('more-itertools', '==5.0.0'),
('oauthlib', '==3.0.1'),
('pluggy', '==0.8.1'),
('py', '==1.7.0'),
('pyasn1', '==0.4.5'),
('pyasn1-modules', '==0.2.4'),
('pycparser', '==2.19'),
('pyjwt', '==1.7.1'),
('pytest', '==4.2.0'),
('pytest-cov', '==2.6.1'),
('pytest-forked', '==1.0.1'),
('pytest-xdist', '==1.26.1'),
('python-dateutil', '==2.8.0'),
('pyyaml', '==3.13'),
('requests', '==2.21.0'),
('requests-oauthlib', '==1.2.0'),
('rsa', '==4.0'),
('six', '==1.12.0'),
('urllib3', '==1.24.1'),
('websocket-client', '==0.54.0')]
It seems like a regression to me, but I may be wrong. Could you guys help me out figuring this out?
I'm using Python v3.7.2
from python.
nevermind! Found a fix!
For those looking and reaching this issue, I found the fix looking here:
#409
and here
#526
from python.
Related Issues (20)
- Test 1
- K8s Python interface issue HOT 2
- Manage resourceVersion to allow resilient restart of watch method HOT 2
- [stream/ws_client] Reading long stdout is truncated at 32768 chars HOT 3
- Request timeout in APIs HOT 9
- Watch() returns wrong "DELETED" event HOT 2
- Version 30.0? HOT 6
- Failed to create deployment HOT 2
- Kubernetes Async API Calls with Callbacks Support HOT 9
- How to get access to the slack channel? HOT 2
- Watch stream missing job completion events HOT 1
- `KubeConfigMerger` object poorly structured when `KUBECONFIG` is a list of files. HOT 2
- Cannot change kube config at runtime HOT 4
- k8s_sdk29.0 error:no attribute 'BatchV1beta1Api' HOT 2
- Support new Kubernetes Sidecar Containers HOT 2
- When I call the query deployment interface, it returns 403 forbidden. HOT 1
- Make google-auth an optional dependency
- kubernetes.client.AuthenticationV1TokenRequest() fails HOT 1
- Allow utils.create_from_yaml to use server side apply
- Error with api when set configuration assert_hostname as true
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from python.