abhay06102003/uptrain

Your open-source ML monitoring and refinement toolkit.

Open-source, self-hosted, easy-to-configure tool to improve ML models in production.

UpTrain is an open-source, data-secure tool for ML practitioners to observe and refine their ML models by monitoring their performance, checking for (data) distribution shifts, and collecting edge cases to retrain them upon. It integrates seamlessly with your existing production pipelines and takes minutes to get started ⚡.

Key Features 💡

• Data Drift Checks - identify distribution shifts in your model inputs.
• Performance Monitoring - track the performance of your models in realtime and get alerted as soon as a dip is observed.
• Edge Case Signals - user-defined signals and statistical techniques to detect out-of-distribution data-points.
• Data Integrity Checks - checks for missing or inconsistent data, duplicate records, data quality, etc.
• Customizable metrics - define custom metrics that make sense for your use case.
• Automated Retraining - automate model retraining by attaching your training and inference pipelines.
• Model Bias - track popularity bias in your recommendation models.
• Data Security - your data never goes out of your machine.

🚨Coming soon🚨

• Realtime Dashboards - to visualize your model's health.
• Embeddings Support - specialized dashboards to understand model-inferred embeddings.
• Slack Integration - get alerts on Slack.
• Label Shfit - identify drifts in your predictions. Specially useful in cases when ground truth is unavailable.
• Prediction Stability - filter cases where model prediction is not stable.
• AI Explainability - understand relative importance of multiple features on predictions.
• Adversarial Checks - combat adversarial attacks

And more.

Get started 🙌

Install the package through pip:

pip install uptrain

Run your first example:

git clone [email protected]:uptrain-ai/uptrain.git
cd uptrain/examples/1_orientation_classification
pip install jupyterlab
jupyter lab

For more info, visit our get started guide.

Why UpTrain 🤔?

Machine learning (ML) models are widely used to make critical business decisions. Still, no ML model is 100% accurate, and, further, their accuracy deteriorates over time 😣. For example, Sales prediction becomes inaccurate over time due to a shift in consumer buying habits. Additionally, due to the black boxiness nature of ML models, it's challenging to identify and fix their problems.

UpTrain solves this. We make it easy for data scientists and ML engineers to understand where their models are going wrong and help them fix them before others complain 🗣️.

UpTrain can be used for a wide variety of Machine learning models such as LLMs, recommendation models, prediction models, Computer vision models, etc.

We are constantly working to make UpTrain better. Want a new feature or need any integrations? Feel free to create an issue or contribute directly to the repository.

License 💻

This repo is published under Apache 2.0 license. We're currently focused on developing non-enterprise offerings that should cover most use cases. In the future, we will add a hosted version which we might charge for.

Stay Updated ☎️

We are continuously adding tons of features and use cases. Please support us by giving the project a star ⭐!

Provide feedback (Harsher the better 😉)

We are building UpTrain in public. Help us improve by giving your feedback here