Exec calls

[kubernetes-client]

Exec calls

[sebgoa]

You can do exec over websockets instead of SPDY. So would be good to investigate websockets instead of the sdpy route (which will be osbolete anyway).

[dims]

@mbohlool @mrmcmuffinz : i have fixed the connect_post_namespaced_pod_exec in #123 - added a e2e test as well to show that it works fine and we don't regress when we generate new client from swagger.

[mbohlool]

assert_hostname is not in config package, I understand the confusion though, we should clear that up.

on a side note, you don't need to pass kube config location if it is in the default location (as I see it is for your case).

[mbohlool]

Well I don't mind creating a new issue for post work. The access to stderr and stdout separately was one of my comments on original PR too. The majority part of the task is done so I will leave it to you and @dims if you want to close this and create a new issue, but I don't think our work on exec calls is done until we solve those issues. I would like to have a similar approach to Watch here. Will try something today.

[mbohlool]

cc @caesarxuchao

[mbohlool]

realted: http://stackoverflow.com/questions/37349440/upgrade-request-required-when-running-exec-in-kubernetes

I guess if we change code generator to accept headers, it should be possible to pass upgrade headers to the call.

Update: Look like we need an SPDY client for python too.

[diegodelemos]

Is it feasible to have it soon? It would be nice to have a date in order to know if I can use it within my deadline. If not, do you have some workaround in mind? Thank you in advance.

[mbohlool]

I don't have a date yet, the problem is urllib3 does not support SPDY protocol. we need to use something like python-spdylay. Unless somebody own this (or I found the time to look at it in more depth) I can't put a date on it.

[mbohlool]

Just spend a little more time with python-spdylay. Even their client example is not working and it does not seem to be actively developed/supported. If anybody have any idea of how to talk SPDY in python, please let me know.

[mbohlool]

More information here: kubernetes/kubernetes#24668

[diegodelemos]

Moreover, there is kubernetes/kubernetes#7452 which stands that SPDY is deprecated and HTTP/2 might be the right option, but looks like it is still hanging in the air. Anyways, it is SPDY what is working for current versions so we need to somehow find a workaround.

[mbohlool]

A little more digging pointed me to a library called thor. I will give it a try. If you can try it too to see if you can establish a SPDY connection with it, that would be nice. My assumption is SPDY is always secure so there should be a way to specify client-cert and other cert files that we normally set in kubernetes.client.configuration class.

[mbohlool]

Thanks @sebgoa it look like websockets are supported as a fall-back: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api-conventions.md#websockets-and-spdy

[mrmcmuffinz]

@mbohlool is there an example that you could provide demonstrating the use of exec via websockets?

[mbohlool]

@mrmcmuffinz I did not have time to do it. That is why I added "Help needed" to this issue so somebody from the community hopefully help.

[mrmcmuffinz]

@mbohlool My apologies, I thought you had some example. Do you happen to know who we can talk to, to help out? Can we work out defining what exactly it takes to get this done?

[mrmcmuffinz]

Adding my stackoverflow question here too, for other people who stumble across this. Also has anyone looked into speaking with the developers of urilib3 to see if they will support http/2? That way there is a minimum impact to the api_client...

[mbohlool]

The ApiException for exec or any call needs SPDY/Websockets is informative. The right way to fix this is to catch that exception in api_client.py and start a websocket call instead. For the first step, try to write a python code that does not use this repo, and directly call into exec REST API using websocket. use kubectl proxy to get rid of authorization part and call into local host for simplification.

[mrmcmuffinz]

I actually tried doing that but I could not figure out how to setup the authentication part of the code for the websocket. I have three pem certificates that were given to me and I'm just not sure how that fits into the picture. Thank you for your response btw.

[mbohlool]

have you try kubectl proxy? that way you should not worry about authentication for now. it will proxy your call from localhost with the right authentication headers to the remote host.

[mrmcmuffinz]

Ohh wow I totally forgot about the kubectl proxy command, if I do that should I switch my url to localhost in the websocket url versus the actual endpoint itself?

Edit:
nvm, I reread again and that is indeed what you suggested I do.

[mbohlool]

Yes. lets have a working code with proxy then think about how to incorporate it into the client-python.

[mrmcmuffinz]

Yeah I think I'm a bit further however I get a connection reset by peer error right now. I think I will need to read up on sockets in python as well as what all I need to do in order to setup the appropriate message for communicating. FYI: I'm using a package called hyper for the actual communication unless you can suggest anything better.

[mrmcmuffinz]

Here is the issue against urllib3 for http/2 suuport urllib3/urllib3#836.

[chekolyn]

@mrmcmuffinz I was wondering how you are testing this. I was doing some basic testing with hyper; but the exec call needs a "SPDY/3.1" upgrade header that hyper doesn't know how to handle.

If you do a connection upgrade with "SPDY/3.1" you will get this exception: https://github.com/Lukasa/hyper/blob/24b3b37d226bedb6199f22b788e066b91dbf892a/hyper/common/conncetion.py#L130

And if you do an h2 upgrade you will get this from kubernetes (since http2 is not implemented yet):
https://github.com/kubernetes/kubernetes/blob/55bee3ad21f025b1416a4e1f10de753f484b66d3/pkg/util/httpstream/spdy/upgrade.go#L48

[mrmcmuffinz]

Hi @chekolyn in all honesty I put it on hold and started looking at other clients. I just wanted to use these python client with py.test for FVT but that kinda went out the door because of this limitation.

Edit:
As for the socket workaround that was another rabbit hole that I did not want to go into right now.

[chekolyn]

OK guys a gave websockets a try and I have a very limited POC

I can confirm websockets works!
(Tested on OpenShift)

# POC for Websocket exec calls to pods:
#
# Channel 0 is STDIN, 1 is STDOUT, 2 is STDERR (if TTY is not requested),
# and 3 is a special error channel. The server only reads from STDIN,
# writes to the other three.

import websocket
import ssl
from subprocess import check_output


def on_message(ws, message):
    print 'message received ..'
    print message


def on_error(ws, error):
    print 'error happened .. '
    print error


def on_close(ws):
    print "### closed ###"


def on_open(ws):

    print 'Opening Websocket connection to the server ... '
    # This session_key I got, need to be passed over websocket header isntad
    # of ws.send.
    #ws.send(session_key)

token = check_output(["oc", "whoami", "-t"])
pod = "router-1-8u540"
exec_command = "uptime".replace(" ", "+")

url = "wss://localhost:8443/api/v1/namespaces/default/pods/%s/exec?command=%s&stderr=true&stdin=true&stdout=true&tty=false" % (pod, exec_command)
header = "Authorization: Bearer " + token



if __name__ == "__main__":
    websocket.enableTrace(False)
    ws = websocket.WebSocketApp(url,
                                on_message = on_message,
                                on_error = on_error,
                                on_close = on_close,
                                header = [header]
                                )
    ws.on_open = on_open
    ws.run_forever(sslopt={"cert_reqs": ssl.CERT_NONE})

#    ws.on_message = on_message
#    ws.run_forever(sslopt={"cert_reqs": ssl.CERT_NONE})

[dims]

@chekolyn "oc" implies openshift? does this work in a pure kubernetes environment?

[chekolyn]

@dims I haven't tested it on a pure k8s but I believe It should work, the oc command is just grabbing the current auth token to authenticate via headers (perhaps this is the only portion that it's OpenShift specific in the POC, you can probably remove the check_output for a valid token and it should work too ). The url api exec call is pure kubernetes and websockets calls seem to be working fine.

I'm basically using this authentication strategy in the api call: https://kubernetes.io/docs/user-guide/accessing-the-cluster/#without-kubectl-proxy-post-v13x

[mrmcmuffinz]

Hi, has anyone else validated that these changes have indeed solved our exec demands? Do I need to change my code which is included in the stackoverflow link above or should I be good to go?

[mbohlool]

@mrmcmuffinz did you build the client from master? The change just merged into master and will be available in the next release, but you should be able to follow the guide in README file to install client from master and give it a try. Let us know if that worked.

[mrmcmuffinz]

Hi @mbohlool I have not had the chance yet however I found it odd that there was some deliveries but no review comments when I looked at the commits. Also I read that there were no tests delivered as part of the delivery and that the auto generator would potentially overwrite some of the code... I will certainly be taking a look at this over the weekend. Thanks to those who were involved!

[dims]

@mrmcmuffinz : we have an e2e test that actually runs k8s and tests the exec code path - https://github.com/kubernetes-incubator/client-python/blob/master/kubernetes/e2e_test/test_client.py#L75

[mbohlool]

@mrmcmuffinz look at #120 for more accurate information. There are lots of comments and e2e tests. Also we excluded those files from auto-generation. Let me know how was your test.

[mrmcmuffinz]

Yeah sorry for the misleading information, I should have dug a bit deeper, saw this dims@7e9adec, into the commits versus just reading some of the commits I saw as part of the commits above.

[mrmcmuffinz]

I cloned the repo and tried installing the client but it still does not work for me...

[acabrer@mrmcmuffins client-python]$ python2.7 setup.py build
/usr/local/lib/python2.7/site-packages/setuptools/dist.py:340: UserWarning: The version specified ('1.0.0-snapshot') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
  "details." % self.metadata.version
running build
running build_py
running egg_info
writing requirements to kubernetes.egg-info/requires.txt
writing kubernetes.egg-info/PKG-INFO
writing top-level names to kubernetes.egg-info/top_level.txt
writing dependency_links to kubernetes.egg-info/dependency_links.txt
reading manifest file 'kubernetes.egg-info/SOURCES.txt'
writing manifest file 'kubernetes.egg-info/SOURCES.txt'
[91541 refs]

Seems this part was not clear enough:

[acabrer@mrmcmuffins client-python]$ sudo python2.7 setup.py develop
[sudo] password for acabrer: 
/usr/lib/python2.7/site-packages/setuptools/dist.py:294: UserWarning: The version specified ('1.0.0-snapshot') is an invalid version, this may not work as expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440 for more details.
  "details." % self.metadata.version
running develop
running egg_info
writing requirements to kubernetes.egg-info/requires.txt
writing kubernetes.egg-info/PKG-INFO
writing top-level names to kubernetes.egg-info/top_level.txt
writing dependency_links to kubernetes.egg-info/dependency_links.txt
reading manifest file 'kubernetes.egg-info/SOURCES.txt'
writing manifest file 'kubernetes.egg-info/SOURCES.txt'
running build_ext
Creating /usr/lib/python2.7/site-packages/kubernetes.egg-link (link to .)
kubernetes 1.0.0-snapshot is already the active version in easy-install.pth

Installed /home/acabrer/git.workspace/client-python
Processing dependencies for kubernetes===1.0.0-snapshot
Searching for ipaddress==1.0.18
Best match: ipaddress 1.0.18
Adding ipaddress 1.0.18 to easy-install.pth file

Using /usr/lib/python2.7/site-packages
Searching for oauth2client==4.0.0
Best match: oauth2client 4.0.0
Adding oauth2client 4.0.0 to easy-install.pth file

Using /usr/lib/python2.7/site-packages
Searching for PyYAML==3.12
Best match: PyYAML 3.12
Adding PyYAML 3.12 to easy-install.pth file

Using /usr/lib64/python2.7/site-packages
Searching for python-dateutil==2.6.0
Best match: python-dateutil 2.6.0
Processing python_dateutil-2.6.0-py2.7.egg
python-dateutil 2.6.0 is already the active version in easy-install.pth

Using /usr/lib/python2.7/site-packages/python_dateutil-2.6.0-py2.7.egg
Searching for certifi==2016.9.26
Best match: certifi 2016.9.26
Processing certifi-2016.9.26-py2.7.egg
certifi 2016.9.26 is already the active version in easy-install.pth

Using /usr/lib/python2.7/site-packages/certifi-2016.9.26-py2.7.egg
Searching for six==1.10.0
Best match: six 1.10.0
Adding six 1.10.0 to easy-install.pth file

Using /usr/lib/python2.7/site-packages
Searching for urllib3==1.20
Best match: urllib3 1.20
Adding urllib3 1.20 to easy-install.pth file

Using /usr/lib/python2.7/site-packages
Searching for httplib2==0.9.2
Best match: httplib2 0.9.2
Adding httplib2 0.9.2 to easy-install.pth file

Using /usr/lib/python2.7/site-packages
Searching for pyasn1==0.1.9
Best match: pyasn1 0.1.9
Processing pyasn1-0.1.9-py2.7.egg
pyasn1 0.1.9 is already the active version in easy-install.pth

Using /usr/lib/python2.7/site-packages/pyasn1-0.1.9-py2.7.egg
Searching for rsa==3.4.2
Best match: rsa 3.4.2
Adding rsa 3.4.2 to easy-install.pth file
Installing pyrsa-encrypt-bigfile script to /usr/bin
Installing pyrsa-encrypt script to /usr/bin
Installing pyrsa-verify script to /usr/bin
Installing pyrsa-sign script to /usr/bin
Installing pyrsa-priv2pub script to /usr/bin
Installing pyrsa-decrypt script to /usr/bin
Installing pyrsa-decrypt-bigfile script to /usr/bin
Installing pyrsa-keygen script to /usr/bin

Using /usr/lib/python2.7/site-packages
Searching for pyasn1-modules==0.0.8
Best match: pyasn1-modules 0.0.8
Adding pyasn1-modules 0.0.8 to easy-install.pth file

Using /usr/lib/python2.7/site-packages
Finished processing dependencies for kubernetes===1.0.0-snapshot

>>> response = cli.connect_post_namespaced_pod_exec(pod, ns, stderr=True, stdin=True, stdout=True, command=exec_command)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "kubernetes/client/apis/core_v1_api.py", line 3396, in connect_post_namespaced_pod_exec
    (data) = self.connect_post_namespaced_pod_exec_with_http_info(name, namespace, **kwargs)
  File "kubernetes/client/apis/core_v1_api.py", line 3501, in connect_post_namespaced_pod_exec_with_http_info
    collection_formats=collection_formats)
  File "kubernetes/client/api_client.py", line 329, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "kubernetes/client/api_client.py", line 153, in __call_api
    _request_timeout=_request_timeout)
  File "kubernetes/client/api_client.py", line 383, in request
    body=body)
  File "kubernetes/client/rest.py", line 275, in POST
    body=body)
  File "kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Date': 'Thu, 09 Feb 2017 04:23:14 GMT', 'Content-Length': '139', 'Content-Type': 'application/json'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Upgrade request required","reason":"BadRequest","code":400}

Am I missing something?

[acabrer@mrmcmuffins client-python]$ python2.7 -m pip list --format=columns |grep kubernetes
kubernetes                   1.0.0-snapshot     

[mbohlool]

you need to install it not build.

$ python2.7 setup.py install

I suggest you create a clean virtualenv first to make sure you are not conflicting with any other installed package.

[mrmcmuffinz]

I did a develop... but none the less I will just uninstall and reinstall

[dims]

@mrmcmuffinz : If you see the test i pointed out to, you will see connect_get_namespaced_pod_exec works. to get connect_post_namespaced_pod_exec to work, please patch the following line to allow "POST"
https://github.com/kubernetes-incubator/client-python/blob/master/kubernetes/client/api_client.py#L349

[mrmcmuffinz]

Yeah I saw the test later, sorry about that however for your patch suggestion that does not sound like a good idea to me. Perhaps it will get to work in my case however that breaks assumptions such that you forwarded the post request via get...

[mrmcmuffinz]

There is a requirements.txt which I was not aware of until I looked into the commit comments... after installing the requirements file

(kpclient)[acabrer@mrmcmuffins bin]$ python -m pip install -r ~/git.workspace/client-python/requirements.txt 
Requirement already satisfied (use --upgrade to upgrade): certifi>=14.05.14 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/certifi-2017.1.23-py2.7.egg (from -r /home/acabrer/git.workspace/client-python/requirements.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): six>=1.9.0 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/six-1.10.0-py2.7.egg (from -r /home/acabrer/git.workspace/client-python/requirements.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): python-dateutil>=2.5.3 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/python_dateutil-2.6.0-py2.7.egg (from -r /home/acabrer/git.workspace/client-python/requirements.txt (line 3))
Collecting setuptools>=21.0.0 (from -r /home/acabrer/git.workspace/client-python/requirements.txt (line 4))
  Downloading setuptools-34.1.1-py2.py3-none-any.whl (389kB)
    100% |████████████████████████████████| 399kB 1.9MB/s 
Requirement already satisfied (use --upgrade to upgrade): urllib3>=1.19.1 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/urllib3-1.20-py2.7.egg (from -r /home/acabrer/git.workspace/client-python/requirements.txt (line 5))
Requirement already satisfied (use --upgrade to upgrade): pyyaml>=3.12 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/PyYAML-3.12-py2.7-linux-x86_64.egg (from -r /home/acabrer/git.workspace/client-python/requirements.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): oauth2client>=4.0.0 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/oauth2client-4.0.0-py2.7.egg (from -r /home/acabrer/git.workspace/client-python/requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): ipaddress>=1.0.17 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/ipaddress-1.0.18-py2.7.egg (from -r /home/acabrer/git.workspace/client-python/requirements.txt (line 8))
Collecting websocket-client>=0.32.0 (from -r /home/acabrer/git.workspace/client-python/requirements.txt (line 9))
  Downloading websocket_client-0.40.0.tar.gz (196kB)
    100% |████████████████████████████████| 204kB 2.5MB/s 
Collecting packaging>=16.8 (from setuptools>=21.0.0->-r /home/acabrer/git.workspace/client-python/requirements.txt (line 4))
  Downloading packaging-16.8-py2.py3-none-any.whl
Collecting appdirs>=1.4.0 (from setuptools>=21.0.0->-r /home/acabrer/git.workspace/client-python/requirements.txt (line 4))
  Downloading appdirs-1.4.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): httplib2>=0.9.1 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/httplib2-0.10.3-py2.7.egg (from oauth2client>=4.0.0->-r /home/acabrer/git.workspace/client-python/requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): pyasn1>=0.1.7 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/pyasn1-0.2.2-py2.7.egg (from oauth2client>=4.0.0->-r /home/acabrer/git.workspace/client-python/requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): pyasn1-modules>=0.0.5 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/pyasn1_modules-0.0.8-py2.7.egg (from oauth2client>=4.0.0->-r /home/acabrer/git.workspace/client-python/requirements.txt (line 7))
Requirement already satisfied (use --upgrade to upgrade): rsa>=3.1.4 in /home/acabrer/virt.env.py/kpclient/lib/python2.7/site-packages/rsa-3.4.2-py2.7.egg (from oauth2client>=4.0.0->-r /home/acabrer/git.workspace/client-python/requirements.txt (line 7))
Collecting backports.ssl-match-hostname (from websocket-client>=0.32.0->-r /home/acabrer/git.workspace/client-python/requirements.txt (line 9))
Collecting pyparsing (from packaging>=16.8->setuptools>=21.0.0->-r /home/acabrer/git.workspace/client-python/requirements.txt (line 4))
  Downloading pyparsing-2.1.10-py2.py3-none-any.whl (56kB)
    100% |████████████████████████████████| 61kB 4.5MB/s 
Installing collected packages: pyparsing, packaging, appdirs, setuptools, backports.ssl-match-hostname, websocket-client
  Found existing installation: setuptools 20.3
    Uninstalling setuptools-20.3:
      Successfully uninstalled setuptools-20.3
  Running setup.py install for websocket-client ... done
Successfully installed appdirs-1.4.0 backports.ssl-match-hostname-3.5.0.1 packaging-16.8 pyparsing-2.1.10 setuptools-34.1.1 websocket-client-0.40.0
You are using pip version 8.1.1, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

I get the below:

(kpclient)[acabrer@mrmcmuffins bin]$ ~/kube.py 
[kube.py:25 - main() ] (0)
Reason: hostname 'x.x.x.x' doesn't match u'10.10.10.1'

I'm guessing this is because I need to do the proxy command in kubectl....

[mrmcmuffinz]

Hi @dims thanks for the update, I have pulled down the latest code and installed it however I still have the below error

(kpclient)[acabrer@mrmcmuffins bin]$ ~/kube.py 
[kube.py:25 - main() ] (0)
Reason: hostname 'x.x.x.x' doesn't match u'10.10.10.1'
Traceback (most recent call last):
  File "/home/acabrer/kube.py", line 22, in main
    response = cli.connect_post_namespaced_pod_exec(pod, ns, stderr=True, stdin=True, stdout=True, command=exec_command, tty=False)
  File "build/bdist.linux-x86_64/egg/kubernetes/client/apis/core_v1_api.py", line 3396, in connect_post_namespaced_pod_exec
    (data) = self.connect_post_namespaced_pod_exec_with_http_info(name, namespace, **kwargs)
  File "build/bdist.linux-x86_64/egg/kubernetes/client/apis/core_v1_api.py", line 3501, in connect_post_namespaced_pod_exec_with_http_info
    collection_formats=collection_formats)
  File "build/bdist.linux-x86_64/egg/kubernetes/client/api_client.py", line 329, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "build/bdist.linux-x86_64/egg/kubernetes/client/api_client.py", line 153, in __call_api
    _request_timeout=_request_timeout)
  File "build/bdist.linux-x86_64/egg/kubernetes/client/api_client.py", line 354, in request
    headers=headers)
  File "build/bdist.linux-x86_64/egg/kubernetes/client/ws_client.py", line 112, in GET
    reason='\n'.join([str(error) for error in client.errors])
ApiException: (0)

[mbohlool]

try setting configuration.assert_hostname to False.

[mrmcmuffinz]

Do you mean kubernetes.config?

[mrmcmuffinz]

@mbohlool just tried that as well still does not work.

[mrmcmuffinz]

@mbohlool
Ok here is what I did, I added the line you suggested to https://github.com/kubernetes-incubator/client-python/blob/master/kubernetes/client/ws_client.py#L107 and it worked for me. However I had set that to my config object and it was not taking affect.

I'm just running a pwd and it looks like it worked for me:

(kpclient)[acabrer@mrmcmuffins client-python]$ ~/kube.py 
'/\n'

Just to add and confirm a bit more, I can
exec_command = ['/bin/sh', '-c', 'cat /etc/centos-release']
on one of my pods.

(kpclient)[acabrer@mrmcmuffins client-python]$ ~/kube.py 
CentOS Linux release 7.3.1611 (Core) 

[mbohlool]

it should be something like:

from kubernetes.client import configuration
from kubernetes.config import load_kube_config

load_kube_config()
configuration.assert_hostname = False

Set the flag after you loaded the config.

[mrmcmuffinz]

I was doing:

from kubernetes.client.rest import ApiException

path_to_config = "/home/acabrer/.kube/config"
config.load_kube_config(config_file=path_to_config)
config.assert_hostname = False
cli = client.CoreV1Api()

I can certainly try your suggestion.

[mrmcmuffinz]

Yeah, I'm going to be writing some py.test fixtures to pass in the location of the configuration hence why I was using the above code. In the code you are suggestion how would one load a config from non-default location, does the load_kube_config() have a optional argument to pass it in?

[mrmcmuffinz]

To answer my own question yes it does https://github.com/kubernetes-incubator/client-python/blob/master/kubernetes/config/kube_config.py#L283

[dims]

@mbohlool - we should close this one out.

[mbohlool]

I still think the way it's implemented is a little hacky (not the implementation itself but the way we integrated it into the api call by checking an specific API endpoint). I would like to keep this issue a little longer so we can look back and clean it up a little. I will take another look tomorrow.

[mrmcmuffinz]

Hi might I bring my two cents. I still think we need to fix the ged and post requests for exec as they both go down the same path. Lastly I have created a dozen or so tests for the product I'm working on but there is one thing that I don't understand, where does one access the stderr, stdout? All we get back is a response message but those elements are not accessible in any specific manner. I think the api is a bit misleading though perhaps I'm misinterpreting it. However that being said this Git issue was opened to add exec support via websockets and @dims did an excellent job at doing that, despite me reporting some post work needed. Which leads me to believe that what you(@mbohol) are asking for is beyond the extent of the original scope for this item.

[mrmcmuffinz]

I'm ok with either approach @mbohlool and will provide my feedback along the way as now I'm kinda a "stakeholder" in this. Any idea when we will have a new release with these changes, or at least a git tag for marking this particular milestone?

[mbohlool]

I have a WIP PR that will send out soon. Only thing that is not working is stdin. stdout and stderr is working as well as streaming output. I will do another beta release after that PR.

[mrmcmuffinz]

Ok, please keep me posted.

[mbohlool]

@mrmcmuffinz I sent a partial PR. stdin is not working yet (and I am not sure why). You also show concern about get/post going the same path, what is your concern? as far as I can tell websocket does not have a method like http. so get/post is the same for websocket.

[mrmcmuffinz]

I sent a partial PR. stdin is not working yet
Do you need some review on this or are you fine?

What is your concern for get/post going down the same path?
Yeah I realized later and came to the same conclusion, which was that the websocket protocol does not deal with get and post requests however the intent I believe was to have websockets as a failback mechanism to the rest api and what we have right now is it just default for exec for get/post. I think we should have that separated out such that if it is get, it goes down the get path and vice versa for post. That makes it easier to fix when we the refactor comes into play. At least that is my two cents on that point.

Edit 1:
I looked at some of the code you wrote and have a few questions:

1. I'd like to suggest first if we can agree on below before we come up with some implementation?

File descriptor stderr standard error or stream 2
File descriptor stdout standard out or stream 1
File descriptor stdin standard input or input stream.

2. Can we agree that the way the above std streams get externalize, they should be externalized via one way for both websockets and rest?

3. As for stdout and stderr I would like to be able to access those in the response body via some member, example below, do you think this right?

response.stdout
response.stderr

4. Does it really make sense to exertnalize stdin right now and if it does what does that mean for exec calls?

5. Lastly, how do I access the return code of the exec command I just executed?

[mbohlool]

The normal HTTP call (that I assume you mean that by rest) for exec does not work and will fail for exec call.writing to stdin would allow interaction with pod. It is much like kubctl exec command when you attach to stdin.

[zhenhua]

@mbohlool I meet the following error when trying to execute a command in the pod.

 	Exception when calling connect_post_namespaced_pod_exec: (0)
 	Reason: 'NoneType' object is not iterable

I tried both k8s 1.2 and 1.5 cluster. It seems the websockets is not accessable so it fails when calling WebSocket.connect() method. How to enable websockets on the k8s master node?

[mbohlool]

I don't think the issue is the target cluster (I don't think websocket can be disabled, it is always enabled). Would you share your code (fragment)? This can be a bug in our implementation.

[mrmcmuffinz]

@mbohlool I actually edited my prior reply and not sure if you saw the full message could you kindly please revisit it?

@zhenhua Could you kindly please include apart from the fragment/redacted source code but also the stack trace?

[mbohlool]

@mrmcmuffinz

1. That is what I have in PR, right?
2. what do you mean by rest? you mean normal http call? we don't support that. if you mean a blocking call then this question is the same as 3?
3. Good idea.
4. kubectl let you connect to a container using exec (I think the option is -stdin that Passes stdin to the container/pod). This let you interact with container like the last part of my example (that is not working right now and I disabled it)
5. By return code you mean return code of the command you ran on the container? You should somehow output it and access it from the output (something like echo $?).

[zhenhua]

My test code is straight forward enough. Here it is

def _exec_pod_cmd(cli, pod_name, command):
    try:
        resp = cli.connect_post_namespaced_pod_exec(
                    name=pod_name, namespace=namespace, command=command)
        return resp 
    except ApiException as e:
        print("Exception when calling connect_post_namespaced_pod_exec: %s\n" % e)

config.load_kube_config(config_file=path_to_config)
config.assert_hostname = False
cli = client.CoreV1Api()

resp = _list_namespaced_pod(cli, str("name=%s" % rc_name))
pod_name = resp.items[0].metadata.name
print("pod name %s " % pod_name)
resp = _exec_pod_cmd(cli, pod_name, "pwd")
pprint(resp)

Here is some debug trace:

(Pdb) b 108
Breakpoint 5 at /usr/lib/python2.7/site-packages/kubernetes/client/ws_client.py:108
(Pdb) c
> /usr/lib/python2.7/site-packages/kubernetes/client/ws_client.py(108)GET()
-> client = WSClient(configuration, url, headers)
(Pdb) p url
'ws://10.13.10.63:8080/api/v1/namespaces/289-ns/pods/test-rc-jd2rp/exec&command=pwd'
(Pdb) p configuration
<kubernetes.client.configuration.ConfigurationObject object at 0x3012290>
(Pdb) p headers
{'Content-Type': 'application/json', 'Accept': '*/*', 'User-Agent': 'Swagger-Codegen/1.0.0-snapshot/python'}
(Pdb) 

After invoking websocket

(Pdb) b 181
Breakpoint 7 at /usr/lib/python2.7/site-packages/websocket/_app.py:181
(Pdb) c
> /usr/lib/python2.7/site-packages/websocket/_app.py(181)run_forever()
-> host=host, origin=origin)
(Pdb) l
176                 self.sock.connect(self.url, header=self.header, cookie=self.cookie,
177                     http_proxy_host=http_proxy_host,
178                     http_proxy_port=http_proxy_port,
179                     http_no_proxy=http_no_proxy, http_proxy_auth=http_proxy_auth,
180                     subprotocols=self.subprotocols,
181 B->                 host=host, origin=origin)
182                 self._callback(self.on_open)
183     
184                 if ping_interval:
185                     event = threading.Event()
186                     thread = threading.Thread(target=self._send_ping, args=(ping_interval, event))
(Pdb) p self.url
'ws://10.13.10.63:8080/api/v1/namespaces/289-ns/pods/test-rc-jd2rp/exec&command=pwd'
(Pdb) p http_proxy_host
None
(Pdb) p self.header
None
(Pdb) n
TypeError: TypeErro...erable",)
> /usr/lib/python2.7/site-packages/websocket/_app.py(181)run_forever()
-> host=host, origin=origin)
(Pdb) n
> /usr/lib/python2.7/site-packages/websocket/_app.py(219)run_forever()
-> except (Exception, KeyboardInterrupt, SystemExit) as e:
(Pdb) p Exception
<type 'exceptions.Exception'>
(Pdb) n
> /usr/lib/python2.7/site-packages/websocket/_app.py(220)run_forever()
-> self._callback(self.on_error, e)
(Pdb) p e
TypeError("'NoneType' object is not iterable",)
(Pdb) 

Let me ask a stupid question: How to access the websocket, port 8443 or port 8080?

[mbohlool]

@mrmcmuffinz: for no 3. It is a good idea, however API spec defined the return type as string. If you want to get stdout and stderr separately you need to call the API with _preload_content=False and read from each channel separately.

resp = api.connect_get_namespaced_pod_exec(name, 'default',
                                           command=exec_command,
                                           stderr=True, stdin=True,
                                           stdout=True, tty=False,
                                           _preload_content=False)
resp.run_forever()
if resp.peek_stdout(): print("STDOUT: ", resp.read_stdout())
if resp.peek_stderr(): print("STDERR: ", resp.read_stderr())

[mbohlool]

@zhenhua, I guess your problem is header. it should not be None. It is fixed as part of my PR.

[zhenhua]

@mbohlool, thanks. I use your mbohlool:exec branch to verify it on my side and it works fine. Only one thing is that, the container parameter of connect_post_namespaced_pod_exec() call is NOT optional, otherwise, it will return an 404 error of 'HTTP/1.1 404 Not Found'.

[mbohlool]

@zhenhua documentation of the parameter says: "Defaults to only container if there is only one container in the pod." I think in case of our examples and e2e tests we have one container but you may have more than one container in your pod. Happy that it worked, I will merge the PR and cut another beta release.

[zhenhua]

@mbohlool You are right. Actually, when creating a pod in my test, there're always have at least two containers, one is registry.access.redhat.com/rhel7/pod-infrastructure:latest and another is my test container. So I have to always specify the container name.

$ docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

63aa2b961a10 registry.v2.service.xxx.org/dongyue/xxx-base-centos7:20170111 "/bin/sh -c /usr/sbin" 40 minutes ago Up 40 minutes k8s_test-rc.12ee117e_test-rc-t4052_289-ns_0b32083c-f806-11e6-b075-c81f66cd4467_1c38b8b6

be4cc79af38c registry.access.redhat.com/rhel7/pod-infrastructure:latest "/pod" 40 minutes ago Up 40 minutes k8s_POD.ae8ee9ac_test-rc-t4052_289-ns_0b32083c-f806-11e6-b075-c81f66cd4467_ffa70144

[ling2yt]

hi,
i test with websocket, but got error message "x509: certificate is valid for 127.0.0.1, not xx.xx.xx.xx"
i use golang like :

wsurl := "wss://xx.xx.xx.xx:8080/r/projects/1a92/kubernetes:6443/api/v1/namespaces/NM/pods/testPod-546cdd8d79-7h8nv/exec?command=ls&container=testPod&stderr=true&stdin=true&stdout=true&tty=false"

u, err := neturl.Parse(wsurl)

rawConn, err := net.Dial("tcp", u.Host)

wsHeaders := http.Header{
        "Authorization":                   {"Bearer "+env_bearer_token},
        "Origin":                   {"https://xx.xx.xx.xx:8080/r/projects/1a92/kubernetes:6443"},
        "Sec-WebSocket-Extensions": {"permessage-deflate; client_max_window_bits, x-webkit-deflate-frame"},

    }

wsConn, resp, err := websocket.NewClient(rawConn, u, wsHeaders, 1024, 1024)

anything wrong ? thanks~

@mbohlool

[lucid-at-dream]

Hi! I'm getting this same error now in version 8.0.1

Here's the offending code:

            v1 = client.CoreV1Api()
            all_pods = [pod.metadata.name for pod in v1.list_namespaced_pod(self.namespace).items]
            pod = next(pod for pod in all_pods if self.name in pod)

            api_response = v1.connect_get_namespaced_pod_exec(
                pod, self.namespace,
                container=self.pod_template.containers[0].name,
                command=['ps', 'axu'],
                stdin=False, stdout=True, stderr=True,
                tty=False
            )

And here's the exception that is being raised:

self = <kubernetes.client.rest.RESTClientObject object at 0x7f870f136cc0>, method = 'GET', url = 'https://****************:443/api/v1/namespaces/elastictests-ozxuefet/pods/test-elastic-cluster-77cf4bfdf9-rsslx/exec'
query_params = [('command', ['ps', 'axu']), ('container', 'elastic-0'), ('stderr', True), ('stdin', False), ('stdout', True), ('tty', False)]
headers = {'Accept': '*/*', 'Content-Type': 'application/json', 'User-Agent': 'Swagger-Codegen/8.0.1/python', 'authorization': 'Bearer f725ca479019a546e359f8a5b7ce9b9d'}, body = None, post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True, _request_timeout=None):
        """
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will be returned without
                                 reading/decoding response data. Default is True.
        :param _request_timeout: timeout setting for this request. If one number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', 'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if PY3 else (int, long)):
                timeout = urllib3.Timeout(total=_request_timeout)
            elif isinstance(_request_timeout, tuple) and len(_request_timeout) == 2:
                timeout = urllib3.Timeout(connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(method, url,
                                                  body=request_body,
                                                  preload_content=_preload_content,
                                                  timeout=timeout,
                                                  headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':
                    r = self.pool_manager.request(method, url,
                                                  fields=post_params,
                                                  encode_multipart=False,
                                                  preload_content=_preload_content,
                                                  timeout=timeout,
                                                  headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct Content-Type
                    # which generated by urllib3 will be overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(method, url,
                                                  fields=post_params,
                                                  encode_multipart=True,
                                                  preload_content=_preload_content,
                                                  timeout=timeout,
                                                  headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is provided
                # in serialized form
                elif isinstance(body, str):
                    request_body = body
                    r = self.pool_manager.request(method, url,
                                                  body=request_body,
                                                  preload_content=_preload_content,
                                                  timeout=timeout,
                                                  headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided arguments.
                             Please check that your arguments match declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.rest.ApiException: (400)
E           Reason: Bad Request
E           HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Thu, 07 Feb 2019 14:03:23 GMT', 'Content-Length': '139'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Upgrade request required","reason":"BadRequest","code":400}

/root/.local/share/virtualenvs/stratio-atf-4Tz-qxfE/lib/python3.7/site-packages/kubernetes/client/rest.py:222: ApiException

I'm using pipenv to manage the working environment. Here are the packages I have installed and corresponding versions:

In [11]: vs = json.loads(open('Pipfile.lock','r').read())
In [12]: [(i, vs['default'][i]['version']) for i in vs['default']]
Out[12]: 
[('adal', '==1.2.1'),
 ('apipkg', '==1.5'),
 ('asn1crypto', '==0.24.0'),
 ('atomicwrites', '==1.3.0'),
 ('attrs', '==18.2.0'),
 ('cachetools', '==3.1.0'),
 ('certifi', '==2018.11.29'),
 ('cffi', '==1.11.5'),
 ('chardet', '==3.0.4'),
 ('coverage', '==4.5.2'),
 ('cryptography', '==2.5'),
 ('elasticsearch', '==6.3.1'),
 ('execnet', '==1.5.0'),
 ('google-auth', '==1.6.2'),
 ('idna', '==2.8'),
 ('kubernetes', '==8.0.1'),
 ('more-itertools', '==5.0.0'),
 ('oauthlib', '==3.0.1'),
 ('pluggy', '==0.8.1'),
 ('py', '==1.7.0'),
 ('pyasn1', '==0.4.5'),
 ('pyasn1-modules', '==0.2.4'),
 ('pycparser', '==2.19'),
 ('pyjwt', '==1.7.1'),
 ('pytest', '==4.2.0'),
 ('pytest-cov', '==2.6.1'),
 ('pytest-forked', '==1.0.1'),
 ('pytest-xdist', '==1.26.1'),
 ('python-dateutil', '==2.8.0'),
 ('pyyaml', '==3.13'),
 ('requests', '==2.21.0'),
 ('requests-oauthlib', '==1.2.0'),
 ('rsa', '==4.0'),
 ('six', '==1.12.0'),
 ('urllib3', '==1.24.1'),
 ('websocket-client', '==0.54.0')]

It seems like a regression to me, but I may be wrong. Could you guys help me out figuring this out?

I'm using Python v3.7.2

[lucid-at-dream]

nevermind! Found a fix!
For those looking and reaching this issue, I found the fix looking here:
#409
and here
#526