TorchServe is a flexible and easy to use tool for serving and scaling PyTorch models in production.
Requires python >= 3.8
curl http://127.0.0.1:8080/predictions/bert -T input.txt# Install dependencies
# cuda is optional
python ./ts_scripts/install_dependencies.py --cuda=cu102
# Latest release
pip install torchserve torch-model-archiver torch-workflow-archiver
# Nightly build
pip install torchserve-nightly torch-model-archiver-nightly torch-workflow-archiver-nightly
# Install dependencies
# cuda is optional
python ./ts_scripts/install_dependencies.py --cuda=cu102
# Latest release
conda install -c pytorch torchserve torch-model-archiver torch-workflow-archiver
# Nightly build
conda install -c pytorch-nightly torchserve torch-model-archiver torch-workflow-archiver
# Latest release
docker pull pytorch/torchserve
# Nightly build
docker pull pytorch/torchserve-nightly
Refer to torchserve docker for details.
- Model Management API: multi model management with optimized worker to model allocation
- Inference API: REST and gRPC support for batched inference
- TorchServe Workflows: deploy complex DAGs with multiple interdependent models
- Default way to serve PyTorch models in
- Export your model for optimized inference. Torchscript out of the box, ORT, IPEX, TensorRT, FasterTransformer
- Performance Guide: builtin support to optimize, benchmark and profile PyTorch and TorchServe performance
- Expressive handlers: An expressive handler architecture that makes it trivial to support inferencing for your usecase with many supported out of the box
- Metrics API: out of box support for system level metrics with Prometheus exports, custom metrics and PyTorch profiler support
- Model Server for PyTorch Documentation: Full documentation
- TorchServe internals: How TorchServe was built
- Contributing guide: How to contribute to TorchServe
- 🤗 HuggingFace Transformers
- Model parallel inference
- MultiModal models with MMF combining text, audio and video
- Dual Neural Machine Translation for a complex workflow DAG
For more examples
We welcome all contributions!
To learn more about how to contribute, see the contributor guide here.
- Grokking Intel CPU PyTorch performance from first principles: a TorchServe case study
- Case Study: Amazon Ads Uses PyTorch and AWS Inferentia to Scale Models for Ads Processing
- Optimize your inference jobs using dynamic batch inference with TorchServe on Amazon SageMaker
- Using AI to bring children's drawings to life
- 🎥 Model Serving in PyTorch
- Evolution of Cresta’s machine learning architecture: Migration to AWS and PyTorch
- 🎥 Explain Like I’m 5: TorchServe
- 🎥 How to Serve PyTorch Models with TorchServe
- How to deploy PyTorch models on Vertex AI
- Quantitative Comparison of Serving Platforms
- Efficient Serverless deployment of PyTorch models on Azure
- Deploy PyTorch models with TorchServe in Azure Machine Learning online endpoints
- Dynaboard moving beyond accuracy to holistic model evaluation in NLP
- A MLOps Tale about operationalising MLFlow and PyTorch
- Operationalize, Scale and Infuse Trust in AI Models using KFServing
- How Wadhwani AI Uses PyTorch To Empower Cotton Farmers
- TorchServe Streamlit Integration
- Dynabench aims to make AI models more robust through distributed human workers
- Announcing TorchServe
Made with contrib.rocks.
This repository is jointly operated and maintained by Amazon, Meta and a number of individual contributors listed in the CONTRIBUTORS file. For questions directed at Meta, please send an email to [email protected]. For questions directed at Amazon, please send an email to [email protected]. For all other questions, please open up an issue in this repository here.
TorchServe acknowledges the Multi Model Server (MMS) project from which it was derived
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent