[Feature] Export optimized cuda graph

[digital-nomad-cheng]

[Feature] Export optimized cuda graph

[yaoyaoding]

Hi @digital-nomad-cheng,

Hidet supports dump and load a model (in the form of a FlowGraph via its save and load method). The dumpped model will store all the information (e.g., model structure, weights) but will leave the compiled kernels to hidet cache.

Currently, Hidet only has a Python-based runtime to execute a flow graph. We are still working on a VM-based runtime (e.g., TVM and Raf has such a VM runtime). In the future, we will have a VM-based runtime that executes a hidet compiled model without Python enviroment, but now we still need python to run the model. Because we used CUDA-graph, the python-based runtime has negligible overhead.

Let me know if you have any other question about Hidet.

[digital-nomad-cheng]

Thanks for clarification, looking forward to the VM-based runtime.