What would it take to support other models like deepseek coder?

[pytorch-labs]

What would it take to support other models like deepseek coder?

[briandw]

I got it working. The main issue what that DeepSeek was using LlamaLinearScalingRotaryEmbedding. I added a scaling factor to the precompute_freqs_cis function and it works! I had to replace the tokenizer and change a few details relating to that.
If I could figure out how to convert the Llama tokenizer to a SentencePiece model file I think that the scaling factor would be the only change needed.

Doesn't anyone know how to go about converting the Llama tokenize to a SentencePiece?

[Chillee]

@briandw The tokenizer interface is pretty simple. https://github.com/pytorch-labs/gpt-fast/blob/main/generate.py#L201

Basically, given a tensor of integers, you need a way of converting it to a string. And given the string, you need a way of converting it to a tensor of integers.

I also think the current tokenizer is already a SentencePieceProcessor? https://github.com/pytorch-labs/gpt-fast/blob/main/generate.py#L285

[briandw]

@Chillee Thanks for your response.
I understand that it's just tokens to id, but I was hoping to be able to use the model without code changes.
LLamaTokeizer isn't an exact replacement, but it's pretty close. I'm just going to use and remove sentence piece and use the LLamaTokenizer.