ValueError: Vocabulary has size 32001 but the model expected a vocabulary of size 32000

[silvacarl2]

ValueError: Vocabulary has size 32001 but the model expected a vocabulary of size 32000

[minhthuc2502]

#1621 The fix is here

[kdcyberdude]

Any updates, I am facing the same problem!

[minhthuc2502]

The model accepts vocab size 32000 tokens but when getting tokens, tokenizer get 32000 tokens + 1 token defined in added_tokens.json. This is the cause of the mismatch in the size. I don't know why with a token added, the vocab size in the config is always 32000. There is some quick fix could be done by ignoring the added token but maybe lead the the quality of model.

[kdcyberdude]

@minhthuc2502 It works, thanks!
This change needs to be done for the Mistral config as well. You can also add a warning message and print the token that is going to be truncated. In my case, it is a <sep> token.

[minhthuc2502]

Thanks for reminding me. Theses models are fine tuned from Llama, so I think the fix on Llama is enough. If there are any model based on Mistral, feel free to open the new issue.