Please help me adding GPT3-Tokenizer

[botisan-ai]

Please help me adding GPT3-Tokenizer

[llegomark]

In case it may be helpful, I have also used the Nutlope twitter bio template, and you can check out how I implemented the tokenizer at https://github.com/llegomark/betterreadings.

To implement the tokenizer, I used the following code:

const tokenizer = new gpt3Tokenizer({ type: 'gpt3' });
const tokens = tokenizer.encode(prompt);
const numTokens = tokens.bpe.length;
const MAX_PROMPT_TOKENS = 400;

if (numTokens > MAX_PROMPT_TOKENS) {
  return new Response(`The prompt has ${numTokens} tokens, which exceeds the maximum limit of ${MAX_PROMPT_TOKENS} tokens.`, {
    status: 400,
    statusText: "Bad Request",
  });
}

I hope this information is helpful to you. Let me know if you have any further questions. Thank you!