[NOTE: This repo is still under development, and any feedback to make it better is welcome 😊 ]
Deploying deep learning models in production could be challenging, as it's far beyond just training models with good performance. As you can see in the following figure, there are several components that need to be properly designed and developed in order to deploy a production level deep learning system:
This repo aims to serve as a an engineering guideline for building production-level deep learning systems to be deployed in real world applications.
The material presented here is mostly borrowed from Full Stack Deep Learning Bootcamp (by Pieter Abbeel at UC Berkeley, Josh Tobin at OpenAI, and Sergey Karayev at Turnitin), TFX workshop by Robert Crowe, and Pipeline.ai's Advanced KubeFlow Meetup by Chris Fregly.
The following figure represent a high level overview of different components in a production level deep learning system:
- Supervised deep learning requires a lot of labeled data
- Labeling own data is costly!
- Here are some resources for data:
- Open source data (good to start with, but not an advantage)
- Data augmentation (a MUST for computer vision, an option for NLP)
- Synthetic data (almost always worth starting with, esp. in NLP)
- Requires: separate software stack (labeling platforms), temporary labor, and QC
- Sources of labor for labeling:
- Crowdsourcing (Mechanical Turk): cheap and scalable, less reliable, needs QC
- Hiring own annotators: less QC needed, expensive, slow to scale
- Data labeling service companies:
- Labeling platforms:
- Prodigy: An annotation tool powered by active learning (by developers of Spacy), text and image
- HIVE: AI as a Service platform for computer vision
- Supervisely: entire computer vision platform
- Labelbox: computer vision
- Scale AI data platform (computer vision & NLP)
- Data storage options:
- Object store: Store binary data (images, sound files, compressed texts)
- Database: Store metadata (file paths, labels, user activity, etc).
- Postgres is the right choice for most of applications, with the best-in-class SQL and great support for unstructured JSON.
- Data Lake: to aggregate features which are not obtainable from database (e.g. logs)
- Feature Store: store, access, and share machine learning features
(Feature extraction could be computationally expensive and nearly impossible to scale, hence re-using features by different models and teams is a key to high performance ML teams).
- FEAST (Google cloud, Open Source)
- Michelangelo Palette (Uber)
- Suggestion: At training time, copy data into a local or networked filesystem (NFS). 1
- It's a "MUST" for deployed ML models:
Deployed ML models are part code, part data. 1 No data versioning means no model versioning. - Data versioning platforms:
- Training data for production models may come from different sources, including Stored data in db and object stores, log processing, and outputs of other classifiers.
- There are dependencies between tasks, each needs to be kicked off after its dependencies are finished. For example, training on new log data, requires a preprocessing step before training.
- Makefiles are not scalable. "Workflow manager"s become pretty essential in this regard.
- Workflow managers:
- Winner language: Python
- Editors:
- Compute recommendations 1:
- For individuals or startups:
- Development: a 4x Turing-architecture PC
- Training/Evaluation: Use the same 4x GPU PC. When running many experiments, either buy shared servers or use cloud instances.
- For large companies:
- Development: Buy a 4x Turing-architecture PC per ML scientist or let them use V100 instances
- Training/Evaluation: Use cloud instances with proper provisioning and handling of failures
- For individuals or startups:
- Cloud Providers:
- GCP: option to connect GPUs to any instance + has TPUs
- AWS:
- Allocating free resources to programs
- Resource management options:
- Unless having a good reason not to, use Tensorflow/Keras or PyTorch. 1
- The following figure shows a comparison between different frameworks on how they stand for "developement" and "production".
- Development, training, and evaluation strategy:
- Always start simple
- Train a small model on a small batch. Only if it works, scale to larger data and models, and hyperparameter tuning!
- Experiment management tools:
- Tensorboard
- provides the visualization and tooling needed for ML experimentation
- Losswise (Monitoring for ML)
- Comet: lets you track code, experiments, and results on ML projects
- Weights & Biases: Record and visualize every detail of your research with easy collaboration
- MLFlow Tracking: for logging parameters, code versions, metrics, and output files, and visualization of the results.
- Always start simple
- Katib: Kubernete's Native System for Hyperparameter Tuning and Neural Architecture Search, inspired by Google vizier and supports multiple ML/DL frameworks (e.g. TensorFlow, MXNet, and PyTorch).
- Hyperas: a simple wrapper around hyperopt for Keras, with a simple template notation to define hyper-parameter ranges to tune.
- SIGOPT: a scalable, enterprise-grade optimization platform
- Ray-Tune: A scalable research platform for distributed model selection (with a focus on deep learning and deep reinforcement learning)
- Sweeps from Weights & Biases: Parameters are not explicitly specified by a developer. Instead they are approximated and learned by a machine learning model.
- Data parallelism: Use it when iteration time is too long (both tensorflow and PyTorch support)
- Model parallelism: when model does not fit on a single GPU
- Other solutions:
- Ray
- Horovod
Machine Learning production software requires a more diverse set of test suites than traditional software:
- Unit and Integration Testing:
- Types of test:
- Training system tests: testing training pipeline
- Validation tests: testing prediction system on validation set
- Functionality tests: testing prediction system on few important examples
- Types of test:
- Continuous Integration: Running tests after each new code change pushed to the repo
- SaaS for continuous integration:
- CircleCI, Travis
- Jenkins, Buildkite
- Consists of a Prediction System and a Serving System
- Prediction System: Process input data, make predictions
- Serving System (Web server):
- Serve prediction with scale in mind
- Use REST API to serve prediction HTTP requests
- Calls the prediction system to respond
- Serving options:
-
- Deploy to VMs, scale by adding instances
-
- Deploy as containers, scale via orchestration
- Containers
- Docker
- Container Orchestration:
- Kubernetes (the most popular now)
- MESOS
- Marathon
-
- Deploy code as a "serverless function"
-
- Deploy via a model serving solution
-
- Model serving:
- Specialized web deployment for ML models
- Batches request for GPU inference
- Frameworks:
- Tensorflow serving
- MXNet Model server
- Clipper (Berkeley)
- SaaS solutions (Seldon, Algorithma)
- Decision making:
- CPU inference:
- CPU inference is preferable if it meets the requirements.
- Scale by adding more servers, or going serverless.
- GPU inference:
- TF serving or Clipper
- Adaptive batching is useful
- CPU inference:
- Transition from monolithic applications towards a distributed microservice architecture could be challenging.
- A Service mesh (consisting of a network of microservices) reduces the complexity of such deployments, and eases the strain on development teams.
- Istio: a service mesh to ease creation of a network of deployed services with load balancing, service-to-service authentication, monitoring, with few or no code changes in service code.
- Purpose of monitoring:
- Alerts for downtime, errors, and distribution shifts
- Catching service and data regressions
- Cloud providers solutions are decent
- Main challenge: memory footprint and compute constraints
- Solutions:
- Quantization
- Reduced model size
- MobileNets
- Knowledge Distillation
- DistillBERT (for NLP)
- Embedded and Mobile Frameworks:
- Tensorflow Lite
- PyTorch Mobile
- Core ML
- ML Kit
- FRITZ
- OpenVINO
- Model Conversion:
- Open Neural Network Exchange (ONNX): open-source format for deep learning models
- Tensorflow Extended (TFX)
- Michelangelo (Uber)
- Google Cloud AI Platform
- Amazon SageMaker
- Neptune
- FLOYD
- Paperspace
- Determined AI
- Domino data lab
- Lessons learned from building practical deep learning systems
- Machine Learning: The High Interest Credit Card of Technical Debt
[1]: Full Stack Deep Learning Bootcamp, Nov 2019.
[2]: Advanced KubeFlow Workshop by Pipeline.ai, 2019.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent