Kubernetes Async API Calls with Callbacks Support

[dirrao]

Kubernetes Async API Calls with Callbacks Support

[tomplus]

Correct me if I'm wrong but you want to have something like this:

create_namespaced_pod("pod-1", "ns-1", cb_success, cb_error)
create_namespaced_pod("pod-2", "ns-2", cb_success, cb_error)

so instead of using async_req i suggest you implement it in this way (draft):

def cb_helper(func_name, cb_success, cb_error, *args, **kwargs):
    try:
       result = func_name(*args, **kwargs)
    except Exception as ex:
       cb_error(ex)
    cb_success(result)

pool = with ThreadPool()
pool.apply_async(cb_helper, create_namespaced_pod, "pod-1", "ns-1", cb_success, cb_error)
pool.apply_async(cb_helper, create_namespaced_pod, "pod-2", "ns-2", cb_success, cb_error)    
....
....

so it uses synchronous requests but from threads so main thread is not blocked...

[dirrao]

I would like to submit the PR if we align on this.

[roycaihw]

Upstream Kubernetes API doesn't support such callback mechanism AFAICT. Could you check with the upstream on this feature request?

[dirrao]

@roycaihw
Async calls are submitted through python thread pool. Python thread pool does support callbacks.

Check api_client.py for the same

        if not async_req:
            return self.__call_api(resource_path, method,
                                   path_params, query_params, header_params,
                                   body, post_params, files,
                                   response_type, auth_settings,
                                   _return_http_data_only, collection_formats,
                                   _preload_content, _request_timeout, _host)

        return self.pool.apply_async(self.__call_api, (resource_path,
                                                       method, path_params,
                                                       query_params,
                                                       header_params, body,
                                                       post_params, files,
                                                       response_type,
                                                       auth_settings,
                                                       _return_http_data_only,
                                                       collection_formats,
                                                       _preload_content,
                                                       _request_timeout,
                                                       _host))

[roycaihw]

CC @tomplus Could you take a look?

[tomplus]

apply_async returns AsyncResult so as you've written it doesn't support callbacks. These functions are auto-generated by OpenAPI Generator so it's not easy to introduce such change.

I can suggest using sync api calls but from a dedicated thread pool where you can add callbacks as you need. What do you think?

[dirrao]

@tomplus
We were avoiding the synchronous requests due to performance reasons. It would be great if we make the requests asyncronous with callbacks support.

[roycaihw]

Marking this as resolved. Please let us know if you have more questions

/close

[k8s-ci-robot]

@roycaihw: Closing this issue.

In response to this:

Marking this as resolved. Please let us know if you have more questions

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.