Comments (5)
Hi @WoosukKwon , thanks for the suggestion, my only concern is the binary size, considering there are many different combinations of python version+cuda version+pytorch version (each wheel is ~500mb), and I used to get warnings because of the large binary size. Did vllm upload all wheels to PyPI?
@zhyncs thanks for your reference:
Perhaps we may temporarily resolve this by using the command such as
PyPI has a unique advantage that other packages can set flashinfer as their dependencies, and I do think it's preferable to upload flashinfer to PyPI.
from flashinfer.
Thanks again for the nice project! Are you interested in uploading the wheels (for CUDA 12.1) to PyPI? This will help users manage the dependency on the FlashInfer library.
@WoosukKwon Perhaps we may temporarily resolve this by using the command such as
pip3 install https://github.com/flashinfer-ai/flashinfer/releases/download/v0.0.2/flashinfer-0.0.2+cu121torch2.1-cp39-cp39-linux_x86_64.whl
@yzh119 If we want to support the workflow of PyPI, we may refer to https://github.com/InternLM/lmdeploy/blob/main/.github/workflows/pypi.yml.
from flashinfer.
@yzh119 I see. What we need at the moment are the Python 3.8-3.11 wheels built for PyTorch 2.1.2 + CUDA 12.1. However, we do agree that maintaining compatibility between the two libraries is quite tricky.
Alternatively, we're currently considering importing FlashInfer as a submodule and building the kernels by ourselves. However, we found that the compilation time of FlashInfer is too long (30+ mins on our machine). Do you have any idea to reduce the time?
from flashinfer.
@yzh119 Also, do you mind if the vLLM team hosts specific PyTorch + CUDA versions of FlashInfer in PyPI under the name of vllm-flashinfer-mirror
or something like that? This will give us more control over the compatibility issue.
from flashinfer.
@yzh119 Also, do you mind if the vLLM team hosts specific PyTorch + CUDA versions of FlashInfer in PyPI under the name of
vllm-flashinfer-mirror
or something like that? This will give us more control over the compatibility issue.
Sounds good.
from flashinfer.
Related Issues (20)
- Does flashinfer support float datatype? HOT 3
- Compare Append Kernel's Results with Xformers HOT 2
- Shared-prefix rope issue HOT 1
- [Install] Build error on main branch
- [LoRA] Roadmap of LoRA operators HOT 1
- Vllm support
- TypeError: get_cu_file_str() missing 1 required positional argument: 'idtype' HOT 1
- Support torch 2.3 HOT 3
- Support MLA (Multi-Head Latent Attention) in DeepSeek-v2 HOT 3
- 能否支持Volta/Tesla架构?
- Compilation fails due to "-Wno-switch-bool" nvcc flag
- multiple definition of `cuda::__3::pipeline...
- Circular import error when importing built-from-source flashinfer HOT 1
- CUDA Error: no kernel image is available for execution on the device (209) /tmp/build-via-sdist-nl8se4dx/flashinfer-0.0.4+cu118torch2.2/include/flashinfer/attention/decode.cuh: line 871 at function cudaFuncSetAttribute(kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, smem_size) HOT 2
- Can BatchDecodeWithPaddedKVCache be used in cascade inference? HOT 1
- Qwen1.5-32B failed: BatchPrefillWithPagedKVCachePyTorchWrapper failed to dispatch group_size 5 HOT 1
- [Feature request] Support attention logits cap with tanh HOT 5
- [Bug report] BatchPrefillWithPagedKVCachePyTorchWrapper failed to dispatch group_size 3 HOT 3
- build raise "cub::BlockAdjacentDifference<__nv_bool, 1024, 1, 1, 860>" has no member "SubtractLeft" HOT 8
- [Q&A] Any palns for different dtypes for Q (query) and KV (kv-cache)? HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flashinfer.