taocao/ab-testing-in-ml

Deploy A/B testing infrastructure in a containerized microservice architecture for Machine Learning applications.

A/B Testing for ML applications

Deploy A/B testing infrastructure in a containerized microservice architecture for Machine Learning applications.

Requirements

This repository uses Kubernetes, helm, ambassador, seldon-core, and seldon-core-analytics.

Getting started

Install prerequisites

1. Install helm

brew install helm

2. Create a Kubernetes cluster: AKS, EKS, GKE or local cluster. For local clusters, one can use minikube, kind, or k3s. In case you are using minikube:

make minikube

Train model

Prepare the model artifacts

make train

Build container images

make build

(minikube only) Load models on minikube's registry

make load

Deploy required components

• ambassador

make ambassador

• seldon-core-analytics

make seldon-core-analytics

• seldon-core

make seldon-core

Deployment

make abtest

Port-forward ambassador

make port

Port-forward Grafana

make port-grafana

Start Streamlit App

make streamlit

References

• Training: https://github.com/SeldonIO/seldon-core/tree/master/examples/models/sklearn_spacy_text
• Seldon-core installation:
  • Install Ambassador: https://docs.seldon.io/projects/seldon-core/en/latest/ingress/ambassador.html
  • Install seldon-core operator: https://docs.seldon.io/projects/seldon-core/en/latest/workflow/install.html
• Seldon-core-analytics installation: https://docs.seldon.io/projects/seldon-core/en/latest/charts/seldon-core-analytics.html
• Custom metrics: https://docs.seldon.io/projects/seldon-core/en/latest/examples/runtime_metrics_tags.html
• Streamlit: https://docs.streamlit.io/library/cheatsheet