How do I replace a spare tokens?

[kouyakamada]

How do I replace a spare tokens?

[ArthurZucker]

cc @itazap that would indeed be a good addition! More and more people pre-allocate some tokens and we don't have a replace token.

[ArthurZucker]

PS: you can already replace directly in the vocab and the added_vocab (since there tokens are part of both)

[lee-onidas]

Hey @ArthurZucker,

I tried replacing a token in the vocab (not the added_tokens) for the tokenizer.json file. But when I try to load the tokenizer back up new_tokenizer = AutoTokenizer.from_pretrained('path/to/tokenizer) I get the following error: "Exception: data did not match any variant of untagged enum ModelWrapper at line 356367 column 3"

Do you know what the problem might be?

[lee-onidas]

Hey @ArthurZucker,

I tried replacing a token in the vocab (not the added_tokens) for the tokenizer.json file. But when I try to load the tokenizer back up new_tokenizer = AutoTokenizer.from_pretrained('path/to/tokenizer) I get the following error: "Exception: data did not match any variant of untagged enum ModelWrapper at line 356367 column 3"

Do you know what the problem might be?

You can ignore this sorry. I found the issue. if you change the vocab in anyway, you need to make sure you also update the merges accordingly.