PROBLEM converting phi2, phi1.5, and phi1 models

[BBC-Esq]

PROBLEM converting phi2, phi1.5, and phi1 models

[vince62s]

Microsoft has changed the config / layers naming in January, ct2 was never updated with this.

https://huggingface.co/microsoft/phi-2/commit/cb2f4533604d8b67de604e7df03bfe6f3ca22869#d2h-639018

[BBC-Esq]

This might be a good opportunity for to try my hand at a pull request with ctranslate2 even though I know zero about C++...but it seems relatively simple to modify the converter for phi...

[BBC-Esq]

For the exactly 2-3 other people in the universe who care about this ;-) here's a link where you can download the older version, but just fyi you have to download the files individually, you can't clone:

https://huggingface.co/microsoft/phi-2/tree/f6ba9c82af0f69e735470b457751c2726e3bbe3c

[BBC-Esq]

Last comment for me for the day...but here's what I get when trying to load the model with the sample llama2 script chat.py that guillikam made:

Traceback (most recent call last):
  File "D:\ctranslate2\chat.py", line 93, in <module>
    main()
  File "D:\ctranslate2\chat.py", line 10, in main
    sp = spm.SentencePieceProcessor(os.path.join(model_dir, "tokenizer.model"))
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\ctranslate2\Lib\site-packages\sentencepiece\__init__.py", line 468, in Init
    self.Load(model_file=model_file, model_proto=model_proto)
  File "D:\ctranslate2\Lib\site-packages\sentencepiece\__init__.py", line 961, in Load
    return self.LoadFromFile(model_file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\ctranslate2\Lib\site-packages\sentencepiece\__init__.py", line 316, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: Not found: "phi-2-ct2-float16\tokenizer.model": No such file or directory Error #2

It seems I was able to convert phi-2, but for whatever reason it didn't create the tokenizer.model in the new folder like it does when converting Mistral and other models...

[edit] I just realized that is way above my experience level...but maybe this'll help??

https://chat.openai.com/share/84e8c2a0-fab5-41d7-a84a-7f195d2f25f2

[minhthuc2502]

when convert the model, you can add this option to save the tokenizer.model in your folder: --copy_files tokenizer.model

[BBC-Esq]

Thanks @minhthuc2502 , but the repository doesn't seem to have a tokenizer.model, which is part of my confusion. BTW, I communicated with the guy who originally did the phi loader/converter, Michael Feil, and he said he doesn't have time to update it to adapt to the changes that Microsoft made to the model in January, 2024. I'd love to try but have no way of visualizing a model's architecture (Netron is mediocre for Pytorch models), and have zero c++ experience. Do you have a sense of if/when the phi loader/converter will be updated by you or someone else? Not a huge deal, just wondering. Anyways, here's where it says no tokenizer.model...I noticed that Michael Feil's ctranslate2 conversion of Phi 1.5 (before Microsoft's change) doesn't have that file...

[minhthuc2502]

It seems like you can't use sentenpiece in chat.py to tokenize the input. You can directly use tokenizer of transformers like: tokenizer = AutoTokenizer.from_pretrained(target_model, trust_remote_code=True).

We have no plans to do an update right now but it would be done if there is time.

[BBC-Esq]

I withdraw this "issue" because phi-2 is too difficult and when it does work, the results are poor...HOWEVER, I strongly recommend updating the scope "StableLM" models that ctranslate2 supports. The new Zephyr is awesome, for example, but there are a slew of other small 3b models that StableLM makes that are best for their sizes IMHO.