torchpipe/torchpipe

An Alternative for Triton Inference Server. Boosting DL Service Throughput 1.5-4x by Ensemble Pipeline Serving with Concurrent CUDA Streams for PyTorch/LibTorch Frontend and TensorRT/CVCUDA, etc., Backends

torchpipe's Issues

Move examples to another repo

Current main repo is around 100 MB when download those examples and the history files. It is better to move those unnecessary files to another repo to keep this one cleaner, and easy to clone.

有多节点多进程的实现么？

只发现了单节点单进程多线程的examples, 有多节点多进程的torchpipe实现么？ 单节点的话， triton的 ensemble 是可以做到多进程单线程pipeline，对于python, 更能充分利用cpu, 只是没有torchpipe多线程轻量, 那比triton的 ensemble的性能好的原因是什么？ 都是cpu 与 gpu拆分成流水线

torchpipe与torchgpipe有什么区别么？

torchpipe与torchgpipe有什么区别么？ 两个库名字很相似

ANNOUNCEMENT: Torchpipe Release v0.4.0

Discussed in #25

^{Originally posted by tp-nan January 12, 2024}

torchpipe 0.4.0 Release Notes

---

torchpipe 0.4.0 is a major release of the library providing multiple new features, and fixes to multiple customer-reported issues.

Release Highlights

torchpipe v0.4.0 includes the following key changes:

• Initial support for CVCUDA 0.5.0 (WITH_CVCUDA=1 pip install -e .).
• Improved support for Python backend and Python filters.
• Added support for TensorRT 9.2.
• Enhanced compatibility and stability.

Compatibility

GPU Compute Capability: 6.1+(7+.x for WITH_CVCUDA)
Ubuntu x86_64: 20.04, 22.04
CUDA Toolkit: 11.0+ (11.2+ for WITH_CVCUDA)
Python: >= 3.7

Known Issues/Limitations

• For the torchpipe installed via whl, the dynamic library path for CVCUDA may need to be set using LD_LIBRARY_PATH

License

torchpipe is licensed under the Apache 2.0 license.

Acknowledgements

Thanks for contributions in various ways to torchpipe

---

This discussion was created from the release Torchpipe Release v0.4.0.

Call for Co-developers: Join our Open Source Project to Improve and Perfect the Codebase

Discussed in #28

^{Originally posted by tp-nan January 30, 2024}
Dear Community,

We are excited to announce that we are seeking passionate co-developers/teams who can help us enhance and perfect our open source project. Our team has been working diligently on this initiative, but we believe that collaboration and diverse perspectives will lead to a better end product. By joining forces with like-minded developers, we aim to create an even more robust, efficient, and user-friendly solution while fostering a vibrant community around it.

Project Overview:

TorchPipe is a multi-instance pipeline parallel library that provides a seamless integration between lower-level acceleration libraries (such as TensorRT and CVCUDA) and RPC frameworks. It achieves strict decoupling from these libraries, offering a thread-safe function interface for the PyTorch frontend at a higher level. At the same time, it enables users to extend the backend capabilities at a lower level. TorchPipe is primarily applied in cloud-based computer vision scenes, with hundreds of millions to billions of daily calls. Its design ensures high service throughput while meeting latency requirements.

Why Contribute?

• Providing Direct Technical Support for you, Prioritizing Optimization of Your Required Scenarios
• Learn new skills and improve existing ones through hands-on practice.
• become co-authers

Responsibilities of Co-developers:

Collaborate effectively with other team members using communication tools.
Actively contribute ideas and solutions during design discussions.
Provide timely feedback on pull requests and issues raised by others.
Adhere to the project's coding standards and guidelines.
Attend regular meetings (if applicable) and actively participate in decision-making processes.

Requirements:

We do not require extensive programming experience from you.

• Familiarity with version control systems, preferably Git.
• A positive attitude and willingness to learn from constructive criticism.
• Self-motivated and able to work independently when necessary.

How to Apply:

Please directly reply to this issue, or apply via email at [email protected]" with the subject line "Application for Co-developer Position." We look forward to hearing from you!

Best Regards,

torchpipe teams