AttributeError: 'HQQLinear' object has no attribute 'weight'

[mxjmtxrm]

AttributeError: 'HQQLinear' object has no attribute 'weight'

[mobicham]

Fixed it, will do a PR right now.

[younesbelkada]

Awesome, thanks !

[younesbelkada]

Hi @mxjmtxrm
Can you share which model are you trying to quantize?

[mxjmtxrm]

It is my own model based on hf llama2 7B. I just modify the bias of qkv proj is True. So the pretrained ckpt contains .bias, and then the above error arised.

[kadirnar]

I use the distil-whisper model. I'm getting the same error. BitsAndBytesConfig optimization method works. HQQ method gives error.

[kadirnar]

@mobicham,

Can you check?

[mobicham]

Can you share a code snippet to reproduce this please?

[younesbelkada]

Thanks everyone ! Indeed I was able to repro with:

from transformers import AutoModelForSpeechSeq2Seq, HqqConfig

model_id = "distil-whisper/distil-large-v2"

quant_config  = HqqConfig(nbits=1, group_size=64, quant_zero=False, quant_scale=False, axis=0)

model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, quantization_config=quant_config, device_map="cuda")
print(model)