boxkite-ml/boxkite

Concept drift monitoring for HA model servers.

{Fast, Correct, Simple} - pick three

Easily compare training and production ML data & model distributions

Goals

Boxkite is an instrumentation library designed from ground up for tracking concept drift in HA (Highly Available) model servers. It integrates well with existing DevOps tools (ie. Grafana, Prometheus, fluentd, kubeflow, etc.), and scales horizontally to multiple replicas with no code or infrastructure change.

• Fast
  • 0.5 seconds to process 1 million data points (training)
  • Sub millisecond p99 latency (serving)
  • Supports sampling for large data sets
• Correct
  • Aggregates histograms from multiple server replicas (using PromQL)
  • Separate counters for discrete and continuous variables (ie. categorical and numeric features)
  • Initialises serving histogram bins from training data set (based on Freedman-Diaconis rule)
  • Handles unseen data, nan, None, inf, and negative values
• Simple
  • One metric for each counter type (no confusion over which metric to choose)
  • Default configuration supports both feature and inference monitoring (easy to setup)
  • Small set of dependencies: prometheus, numpy, and fluentd
  • Extensible metric system (support for image classification coming soon)

Some non-goals of this project are:

• Adversarial detection

If you are interested in alternatives, please refer to our discussions in FAQ.

Getting Started

Follow one of our tutorials to easily get started and see how Boxkite works with other tools:

• Prometheus & Grafana in Docker Compose locally
• Kubeflow & MLflow on Kubernetes with easy online test drive in the browser

See Installation & User Guide for how to use Boxkite in any environment.

FAQ

1. Does boxkite support anomaly / outlier detection?

Prometheus has supported outlier detection in time series data since 2015. Once you've setup KL divergence and K-S test metrics, outlier detection can be configured on top using alerting rules. For a detailed example, refer to this tutorial: https://prometheus.io/blog/2015/06/18/practical-anomaly-detection/.

2. Does boxkite support adversarial detection?

Adversarial detection concerns with identifying single OOD (Out Of Distribution) samples rather than comparing whole distributions. The algorithms are also highly model specific. For these reasons, we do not have plans to support them in boxkite at the moment. As an alternative, you may look into Seldon for such capabilities https://github.com/SeldonIO/alibi-detect#adversarial-detection.

3. Does boxkite support concept drift detection for text / NLP models?

Not yet. This is still an actively researched area that we are keeping an eye on.

4. Does boxkite support tensorflow / pytorch?

Yes, our instrumentation library is framework agnostic. It expects input data to be a list or np.array regardless of how the model is trained.

Contributors

The following people have contributed to the original concept and code

• Han Qiao
• Nguyen Hien Linh
• Luke Marsden
• Mariappan Ramasamy

A full list of contributors, which includes individuals that have contributed entries, can be found here.

Shameless plug

Boxkite is a project from BasisAI, who offer an MLOps Platform called Bedrock.

Bedrock helps data scientists own the end-to-end deployment of machine learning workflows. Boxkite was originally part of the Bedrock client library, but we've spun it out into an open source project so that it's useful for everyone!

boxkite's Issues

Publish to PyPI

• Publish as boxkite
• Add basisai as a collaborator/owner on the PyPI project

Use alert manager in examples to demonstrate slack trigger

Documentation updates

• replace references to bdrk with boxkite in the readme
• move reference to Bedrock at the top to a "this project is from" section at the bottom

plan to get boxkite ready for mass adoption

Boxkite can be a useful tool on its own for sophisticated users, but to get mass adoption we will be more successful if we can show boxkite working well as part of a well-established ecosystem of open source tools. In particular, we want to show somewhat complete "sections" of the MLOps lifecycle with popular MLOps stacks. This will both help existing users of those tools see rapidly how these things can fit together (by following a quick tutorial that shows end-to-end working examples), as well as less mature users looking for complete solutions to be able to "copy and paste" our stack to start using it rapidly.

It's important that the stacks we present are both useful, rapid tutorial stacks as well as being reasonably production-ready.

Additionally, a website showing the value prop clearly and a documentation site with clear and easy to follow tutorials, both with high production value, will help significantly to make the tool appear mature and trustworthy.

I propose that we start with one "best of breed" stack as developing one full example is probably better than several half-baked ones.

Stack to develop

Kubeflow (notebooks + pipelines + Seldon core) + MLflow + Boxkite + Prometheus + Grafana

Deployment modes: local k8s cluster (e.g. minikube), and AWS.
Output format: terraform (targeting either an existing-cluster, or AWS with EKS). This makes it easy to reliably spin up the stack both for testing or targeting AWS for production.

User journey:

• User deploys demo stack using terraform locally or against AWS
• User trains model in a Kubeflow notebook and/or pipeline, using boxkite sdk
• They push the model to MLflow model registry, including the boxkite generated prometheus histogram as an MLflow artifact
• They deploy the model from MLflow to Seldon core, and run a script which sends some inferences against it (the script should send requests with varying distribution over a short timescale, so it demos well)
• They load the Grafana dashboard and are able to live-visualize the changing model drift, and see how they can set alerts against it using Prom alertmanager

Other stacks

Other tools to consider integrating with: ClearML is one of my new favourites.

Also: Pachyderm + Seldon?

Testing

Testing: need to set up CI so that it repeatedly tests the full stack.

Support tracking correlation drift for each feature pair

https://mlops-community.slack.com/archives/C015J2Y9RLM/p1622706500038000?thread_ts=1622473112.442900&cid=C015J2Y9RLM

Update example code to compute inference histogram from train set

https://mlops-community.slack.com/archives/C01TERUGX9U/p1630067772002100

develop documentation

Is that possible to convert the KS test result to p-value?

Hi folks,
Just found this repo and I love the ideal of using histogram for KS test.
But the test result is the D value instead of p-value. Is it possible to convert to p-value so that folks can setup alert more easy?

Copy aisg-demo repo into "examples" directory in boxkite repo

Then update it to work with boxkite python package rather than bdrk (will depend on #4)

Support for PSI and CSI model monitoring metrics

Would be great to see the following metrics available in boxkite!

• Population Stability Index (PSI)
• Characteristic Stability Index (CSI)

Rename this repo `boxkite`

Need someone with more admin privs to do this.

Using box kite with KFServing

When I deploy models using kfserving, is there a way to use boxkite to monitor those models?

develop marketing site

Add unit tests for metric collectors

Feature for pushing metrics to a SaaS service