Error loading gpt3.5 tokenizer

[yujonglee]

Error loading gpt3.5 tokenizer

[yujonglee]

@jonatanklosko Thanks!

For anyone interested in tokenization in Elixir:
https://github.com/fastrepl/fastrepl/blob/da5ab17590f6fa35eccbd9fe7b15b75929eb6655/lib/fastrepl/tokenizer.ex#L10-L19

[seanmor5]

Can you try:

Bumblebee.load_tokenizer({:hf, "Xenova/gpt-3.5-turbo"}, module: GPT2Tokenizer)

?

[yujonglee]

@seanmor5 I tried that, but I got

iex(1)> Bumblebee.load_tokenizer({:hf, "Xenova/gpt-3.5-turbo"}, module: GPT2Tokenizer)
** (ArgumentError) unknown keys [:module] in [module: GPT2Tokenizer], the allowed keys are: [:type]
    (elixir 1.16.2) lib/keyword.ex:359: Keyword.validate!/2
    (bumblebee 0.5.3) lib/bumblebee.ex:896: Bumblebee.load_tokenizer/2
    iex:1: (file)

I am using cb26c2d.

[jonatanklosko]

The API changed, it's no longer a module, try type: :gpt2 : )