Comments (3)
@yzh119 Thanks for the reply. Sounds amazing. Looking forward to the blog post!
from flashinfer.
Hi @WoosukKwon , thanks for your attention.
Could you provide some end-to-end examples (or any documentation) of how to use these kernels? I really want to try
them out on my machine.
Sorry I haven't taken time to do that because of limited bandwidth, documentation is one of the priorities for our official release, I'll add some documentation and provide trying-out examples in README.
Additionally, I've noticed that some of the kernels appear to reimplement existing algorithms like FlashAttention and PagedAttention. Is there any performance benchmark available that compares these to their original implementations? I'm curious about how the kernels perform.
We do have a lot of benchmark results and I'm wrapping up a blog post to describe them. The brief answer is that we are better or on par with existing implementations and I'll explain the performance breakdown analysis in the blog post (Actually, the whole field is evolving so fast and every open-source project keeps improving performance, so it's very important to know how does the performance gain coming from rather than just showing the numbers, and I hope my blog posts would be helpful).
I'm glad that you are interested in this project and I hope we can figure out a set of standard APIs that could benefit a lot of projects. Looking forward to collaborating with you on that if you are interested.
from flashinfer.
Closed as FlashInfer has a nice documentation with examples.
from flashinfer.
Related Issues (20)
- How was the data in the blog measured? HOT 5
- QUESTION: C++ API support Ragged Tensor now? HOT 1
- Does flashinfer support float datatype? HOT 3
- Compare Append Kernel's Results with Xformers HOT 2
- Shared-prefix rope issue HOT 1
- [Install] Build error on main branch
- [LoRA] Roadmap of LoRA operators HOT 1
- Vllm support
- TypeError: get_cu_file_str() missing 1 required positional argument: 'idtype' HOT 1
- Support torch 2.3 HOT 3
- Support MLA (Multi-Head Latent Attention) in DeepSeek-v2 HOT 3
- 能否支持Volta/Tesla架构?
- Compilation fails due to "-Wno-switch-bool" nvcc flag
- multiple definition of `cuda::__3::pipeline...
- Circular import error when importing built-from-source flashinfer HOT 1
- CUDA Error: no kernel image is available for execution on the device (209) /tmp/build-via-sdist-nl8se4dx/flashinfer-0.0.4+cu118torch2.2/include/flashinfer/attention/decode.cuh: line 871 at function cudaFuncSetAttribute(kernel, cudaFuncAttributeMaxDynamicSharedMemorySize, smem_size) HOT 2
- Can BatchDecodeWithPaddedKVCache be used in cascade inference? HOT 1
- Qwen1.5-32B failed: BatchPrefillWithPagedKVCachePyTorchWrapper failed to dispatch group_size 5 HOT 1
- [Feature request] Support attention logits cap with tanh HOT 5
- [Bug report] BatchPrefillWithPagedKVCachePyTorchWrapper failed to dispatch group_size 3 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flashinfer.