Comments (2)
Several of us are super familiar with Hidet developers. In fact, I believe @spectrometerHBH and @yaoyaoding are classmates :-)
Both TVM and Hidet support efficient code generation on CUDA targets (e.g. A100, Jetson Xavier), and both are as performant as existing solutions like cutlass/Triton. There is some difference, for example, older generation of TVM suffers from long tuning time, imperfect search space, lack of support for dynamic shape, etc, but fortunately, the new stack we are relying on doesn't.
On the other hand, MLC LLM aims to democratize local LLM computation, which means the scope is broader and we wanted to support AMD/Intel/Apple GPUs, x86/arm CPUs and other hardware platforms. On those non-NV targets, we found that TVM is the only existing solution that works out of the box. In fact, if you check out the binaries we release, Vulkan is good enough to support CUDA targets for personal use of LLMs.
from mlc-llm.
That’s very helpful, thanks for your response!
from mlc-llm.
Related Issues (20)
- [Question] batchsize of prefill step
- [Bug] InternalError: Check failed: (offset + needed_size <= this->buffer.size) is false: storage allocation failure, attempted to allocate 513024 at offset 0 in region that is 163840bytes HOT 1
- [Question] How to use cpp in project HOT 1
- [Question] How to use function calling in MLCChat Android app? HOT 3
- [Question] Trying to use vulkan version 1.1 library on Qualcomm chip leads to tvm runtime error HOT 4
- [Bug] can run mlc_llm chat on T4 with the model convert with A100 HOT 3
- [Bug] org.apache.tvm.Base$TVMError Check failed: token_id < static_cast<int>(token_table_.size()) HOT 10
- [Question] Can you programmatically clear the kv cache? HOT 2
- [Question] It takes too much time for the first token to be returned after a reqeust issued. HOT 4
- [Question] The Android SDK is too large HOT 4
- [Bug] current available model_libs in dist/lib/libmodel_android.a: [] HOT 4
- [Bug] Threadgroup memory size exceeds the maximum threadgroup memory allowed HOT 4
- [Question] CMake build error, missing variable: _CMAKE_CUDA_WHOLE_FLAG not set HOT 2
- [Bug] "Build Android APP from Source" step2: "mlc_llm package" returns error ""path/to/mlc-llm/android/android/mlc4j" does not exist" HOT 9
- [Bug] fine-tuned model deployed with webllm not working HOT 11
- [Bug] Check failed: (it != type_key2index_.end()) is false: Cannot find type ObjectPath. Did you forget to register the node by TVM_REGISTER_NODE_TYPE HOT 1
- [Bug] Could not find a version that satisfies the requirement during installation of prebuilt packages HOT 4
- [Question] How to calculate the VRAM usage in Android package config? HOT 2
- [Model Request] Gemma 9b and 27b
- [Bug] TypeError: GemmaConfig.__init__() missing 1 required positional argument: 'hidden_activation' HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mlc-llm.