xvyaward/owq

Code for the AAAI 2024 Oral paper "OWQ: Outlier-Aware Weight Quantization for Efficient Fine-Tuning and Inference of Large Language Models".

owq's Issues

NaN ppl when running on Llama2-7b without owq (--wbits 4 --target_bit 4).

Hi Author,

I want to test the ppl of OPTQ without owq on your code, but I get a NaN. Could you tell me what I can do?
The test command line is: "python main.py meta-llama/Llama-2-7b-hf c4 --wbits 4 --target_bit 4"

The console log is as below:

error 58015.34375
Quantizing model.layers.29.self_attn.o_proj
time 2.63
error 3950.656494140625
Quantizing model.layers.29.mlp.gate_proj
time 4.79
error 111899.75
Quantizing model.layers.29.mlp.up_proj
time 4.79
error 101437.4375
Quantizing model.layers.29.mlp.down_proj
time 7.13
error 33238.73046875
Quantizing model.layers.30.self_attn.q_proj
time 2.66
error 116491.953125
Quantizing model.layers.30.self_attn.k_proj
time 2.65
error 87742.4375
Quantizing model.layers.30.self_attn.v_proj
time 2.63
error 64090.1484375
Quantizing model.layers.30.self_attn.o_proj
time 2.65
error 4883.51513671875
Quantizing model.layers.30.mlp.gate_proj
time 4.79
error 115987.59375
Quantizing model.layers.30.mlp.up_proj
time 4.81
error 102952.34375
Quantizing model.layers.30.mlp.down_proj
time 6.98
error nan
Quantizing model.layers.31.self_attn.q_proj
time 2.60
error nan
Quantizing model.layers.31.self_attn.k_proj
time 2.65
error nan
Quantizing model.layers.31.self_attn.v_proj
time 2.65
error nan
Quantizing model.layers.31.self_attn.o_proj
time 2.60
error nan
Quantizing model.layers.31.mlp.gate_proj
time 4.80
error nan
Quantizing model.layers.31.mlp.up_proj
time 4.80
error nan
Quantizing model.layers.31.mlp.down_proj
time 7.07
error nan
Running Time : 1751.8
wikitext2
Evaluating ...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [01:13<00:00,  2.30s/it]
nan
ptb
Evaluating ...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [00:27<00:00,  1.17it/s]
nan
c4
Evaluating ...
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 32/32 [02:01<00:00,  3.80s/it]
nan

Thanks

Failed to reproduce the LLama7B perplexity on the Penn Treebank (PTB) dataset.

Thank you for your excellent work.
Why am I unable to reproduce your perplexity metrics on Penn Treebank (PTB) ?
In OWQ Paper: 12.46
Our reproduce: 56.033756256103516

Here is the output after I execute the "python main.py huggyllama/llama-7b c4 --wbits 3 --target_bit 3.01" command.
Thank you and looking forward to your reply.

wikitext2
Evaluating ...
6.676162242889404
Token indices sequence length is longer than the specified maximum sequence length for this model (106527 > 2048). Running this sequence through the model will result in indexing errors
ptb
Evaluating ...
56.033756256103516
Generating validation split: 45576 examples [00:00, 212472.03 examples/s]
Token indices sequence length is longer than the specified maximum sequence length for this model (612151 > 2048). Running this sequence through the model will result in indexing errors
c4
Evaluating ...
8.551858901977539