microsoft/kfp-event-handler

Kubeflow Pipelines Event Handler

Kubeflow Pipelines Event Handler

This microservice is an extension to Kubeflow Pipelines. It notifies a list of subscribers on the Kubeflow Pipeline Run events.

This repo implements KFP Execution events proposal and will be removed once Kubeflow Pipelines provide native support of the execution events.

When a KFP run starts it submits an Argo Workflow CRD and delegates the pipeline execution to it. Argo Workflow (v2.8.0+) produces K8s events corresponding to the execution stages of the workflow such as "Workflow Started", "Node Succeeded", "Workflow Failed", etc. KFP Event Handler watches those events in its namespace.

Once a new event comes up, the event handler:

• identifies the workflow that produced the event
• invokes Argo Workflow API to get the workflow details (e.g. KFP_RUN_ID, parameters, etc.)
• creates a json message containing the event and workflow details
• iterates over the list of subscribers and sends the message to each of the with a POST request

Each subscriber knows how to parse the event message and how to notify its backend system. An example of a subscriber could be a Teams Chanel Message Sender that sends message cards to Teams, a GitHub Bot that notifies a CI pipeline on the result of KFP execution, an Analytics logger that saves execution data to an analytics database, a KFP runner that starts another pipeline once this one has finished.

If a subscriber needs more details on the KFP run, it may request those details with KFP API before invoking its backend system.

Build and Deploy

• Build and push an image with this Dockerfile
• Update kfp-event-handler.yaml with your image name and tag
• Apply kfp-event-handler.yaml to the namespace where you run your Kubeflow pipelines

Subscribe on Events

To subscribe on KFP events add an entry to kfp-event-handler-config ConfigMap (defined in kfp-event-handler.yaml) with an endpoint URI. The endpoint must accept POST requests with the following payload:

{
 "KfpRunId":"ee1c3f1f-031b-49e9-a7ff-2f7a54c11b95",
 "WorkflowName": "tacos-vs-burritos-vw8g9",
 "NodeName": "dataprocessing",
 "EventType":"WorkflowNodeSucceeded",
 "Message":"Succeeded node tacos-vs-burritos-vw8g9.dataprocessing"
}

Currently supported event types:

Event	Occurs when
WorkflowRunning	KFP Run starts
WorkflowFailed	KFP Run is failed or terminated
WorkflowSucceeded	KFP Run successfully finishes
WorkflowNodeFailed	KFP Step fails
WorkflowNodeSucceeded	KFP Step successfully finishes

Enjoy!

kfp-event-handler's Issues

Unable to process kubeflow pipeline event

I am trying to hook into Kubeflow pipeline events. I am trying with Kubeflow version 1.1.0. But I am unable to receive any events in the event handler service that I have configured (A service inside K8 cluster). I see this log when I install "kfp-event-handler" yaml.

$$$$
E0209 12:38:34.942336 1 reflector.go:156] /tmp/go-app/main.go:64: Failed to list *v1.Event: Get "https://10.100.0.1:443/api/v1/namespaces/admin/events?limit=500&resourceVersion=0": dial tcp 10.100.0.1:443: connect: connection refused
E0209 12:38:35.943070 1 reflector.go:156] /tmp/go-app/main.go:64: Failed to list *v1.Event: Get "https://10.100.0.1:443/api/v1/namespaces/admin/events?limit=500&resourceVersion=0": dial tcp 10.100.0.1:443: connect: connection refused
E0209 12:38:36.943707 1 reflector.go:156] /tmp/go-app/main.go:64: Failed to list *v1.Event: Get "https://10.100.0.1:443/api/v1/namespaces/admin/events?limit=500&resourceVersion=0": dial tcp 10.100.0.1:443: connect: connection refused
$$$$

There are no other logs inside the container. The workflow-controller image version installed in kubeflow cluster is 2.3.0.