xvyaward / owq Goto Github PK
View Code? Open in Web Editor NEWCode for the AAAI 2024 Oral paper "OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models".
Home Page: https://arxiv.org/abs/2306.02272
Code for the AAAI 2024 Oral paper "OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models".
Home Page: https://arxiv.org/abs/2306.02272
Hi Author,
I want to test the ppl of OPTQ without owq on your code, but I get a NaN. Could you tell me what I can do?
The test command line is: "python main.py meta-llama/Llama-2-7b-hf c4 --wbits 4 --target_bit 4"
The console log is as below:
error 58015.34375
Quantizing model.layers.29.self_attn.o_proj
time 2.63
error 3950.656494140625
Quantizing model.layers.29.mlp.gate_proj
time 4.79
error 111899.75
Quantizing model.layers.29.mlp.up_proj
time 4.79
error 101437.4375
Quantizing model.layers.29.mlp.down_proj
time 7.13
error 33238.73046875
Quantizing model.layers.30.self_attn.q_proj
time 2.66
error 116491.953125
Quantizing model.layers.30.self_attn.k_proj
time 2.65
error 87742.4375
Quantizing model.layers.30.self_attn.v_proj
time 2.63
error 64090.1484375
Quantizing model.layers.30.self_attn.o_proj
time 2.65
error 4883.51513671875
Quantizing model.layers.30.mlp.gate_proj
time 4.79
error 115987.59375
Quantizing model.layers.30.mlp.up_proj
time 4.81
error 102952.34375
Quantizing model.layers.30.mlp.down_proj
time 6.98
error nan
Quantizing model.layers.31.self_attn.q_proj
time 2.60
error nan
Quantizing model.layers.31.self_attn.k_proj
time 2.65
error nan
Quantizing model.layers.31.self_attn.v_proj
time 2.65
error nan
Quantizing model.layers.31.self_attn.o_proj
time 2.60
error nan
Quantizing model.layers.31.mlp.gate_proj
time 4.80
error nan
Quantizing model.layers.31.mlp.up_proj
time 4.80
error nan
Quantizing model.layers.31.mlp.down_proj
time 7.07
error nan
Running Time : 1751.8
wikitext2
Evaluating ...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [01:13<00:00, 2.30s/it]
nan
ptb
Evaluating ...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [00:27<00:00, 1.17it/s]
nan
c4
Evaluating ...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [02:01<00:00, 3.80s/it]
nan
Thanks
Thank you for your excellent work.
Why am I unable to reproduce your perplexity metrics on Penn Treebank (PTB) ?
In OWQ Paper: 12.46
Our reproduce: 56.033756256103516
Here is the output after I execute the "python main.py huggyllama/llama-7b c4 --wbits 3 --target_bit 3.01" command.
Thank you and looking forward to your reply.
wikitext2
Evaluating ...
6.676162242889404
Token indices sequence length is longer than the specified maximum sequence length for this model (106527 > 2048). Running this sequence through the model will result in indexing errors
ptb
Evaluating ...
56.033756256103516
Generating validation split: 45576 examples [00:00, 212472.03 examples/s]
Token indices sequence length is longer than the specified maximum sequence length for this model (612151 > 2048). Running this sequence through the model will result in indexing errors
c4
Evaluating ...
8.551858901977539
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.