Real-time inference results

[woosukkwon]

Real-time inference results

[WoosukKwon]

Hi @justlovebarbecue

Currently, the main script only produces pruning masks. To get the actual latency speedup, I applied the masks to the original model and remove the zero-ed parts. That is, I transferred the non-zero model weights to a new smaller dense model. Then I measured its end-to-end GPU latency using GPUTimer in

retraining-free-pruning/utils/timer.py

Line 18 in 493e09a

class GPUTimer:

.

Please let me know if you have further question.

[justlovebarbecue]

Thanks for your reply @WoosukKwon! I missed this part previously. One more question, could you please point out where is the function to load the smaller dense model?

Thanks!

[justlovebarbecue]

I guess, the file generate_lut.py provides a function to load a customized model and test the real-time speed using "GPUTimer"? Am I right?

[WoosukKwon]

Yeah, please check out the code in generate_lut.py.