OpenKiwi always download the tokenizer files for XLMRoberta even if a local path is configured.

[yym6472]

OpenKiwi always download the tokenizer files for XLMRoberta even if a local path is configured.

[captainvera]

Hello @yym6472,

Sorry I took a while to respond.

I tested what you are saying, and you are indeed correct. This version of Openkiwi was initially built when transformers was in version < 2 or 3. The transformers library still had a lot of quirks when working with models other than BERT and a couple others.

This was probably introduced to avoid bugs when loading finetuned versions of these models (like you're trying to do here) as we assumed the tokenizer part wouldn't change.
(Could it be possible that it didn't used to get saved with the model? I really can't recall anymore)

So as of today with transformers in a much more mature state, I'd probably classify this as a bug.
We would really appreciate if you could do a PR to fix it :)

[yym6472]

Thanks for your reply! I have proposed a PR #105 that simply removes the above-mentioned two lines.