from_pretrained torch_dtype DO NOT affect model buffers

[Chandler-Bing]

from_pretrained torch_dtype DO NOT affect model buffers

[ArthurZucker]

Actually we want the inv freq to be in float 32 to keep as much precision as possible.

because in the code we specify the dtype, the default torch dtype is not used

import torch.nn as nn
class myModule(nn.Module):
  def __init__(self):
    super().__init__()
    self.register_buffer("freq", torch.ones(12,12, dtype=torch.float32))
torch.set_default_dtype(torch.bfloat16)
module = myModule()
module.freq.dtype

vs

import torch.nn as nn
class myModule(nn.Module):
  def __init__(self):
    super().__init__()
    self.register_buffer("freq", torch.ones(12,12))
torch.set_default_dtype(torch.bfloat16)
module = myModule()
module.freq.dtype

[ArthurZucker]

Hey!
To answer that question I would need a simple reproducer, as well as the output of transformers-cli env to see which version you are running this one.
AFAIK, buffers should we affected. I recently specifically wanted to keep certain buffers in a precision and it did not work

[Chandler-Bing]

Hey! To answer that question I would need a simple reproducer, as well as the output of transformers-cli env to see which version you are running this one. AFAIK, buffers should we affected. I recently specifically wanted to keep certain buffers in a precision and it did not work

@ArthurZucker
sure, you could get the code here, colab

from  transformers import AutoModelForCausalLM
import torch

model = AutoModelForCausalLM.from_pretrained(
        'Qwen/Qwen1.5-0.5B',torch_dtype=torch.bfloat16)

for name,buffer in model.named_buffers():
    if buffer.dtype==torch.float32:
        print(name,buffer.dtype)

the model is randomly chosen to test.

seems just this buffer rotary_emb.inv_freq still torch.float32

[ArthurZucker]

Closing this as it's affecting buffers that don't specify a dtype, but not others.