RuntimeError: You can't move a model that has some modules offloaded to cpu or disk.

[robinsonmhj]

RuntimeError: You can't move a model that has some modules offloaded to cpu or disk.

[robinsonmhj]

@lekurile can you take a look this issue?

[Phocuser]

I am seeing the same thing in my docker container. When I execute the container without GPU. support, the tokenizer loads.
Error loading model microsoft/Phi-3-mini-128k-instruct: You can't move a model that has some modules offloaded to cpu or disk.

self.tokenizer = AutoTokenizer.from_pretrained(self.model_name,

                                                       device_map="auto",
                                                        torch_dtype="auto",
                                                        use_accelerate=False,
                                                        trust_remote_code=True,
                                                        offload_state_dict=False,  # Disable offloading
                                                        load_in_8bit=False,  # Load in 8-bit precision (optional)
                                                        revision="main"  # Specify the revision to load (optional)
                                                       )

Python 3.12.2
Windows 10
RTX2060