New embedding models not recognized

[thomasdc]

New embedding models not recognized

[dmitry-brazhenko]

Ok, understood. I will add this functionality today.
Thanks for sharing

[thomasdc]

See my first comment.

[dmitry-brazhenko]

Hello @thomasdc !

Thanks for raising an issue.

As far as i understand, embeddings algorithms are caclulated on OpenAI side. This algorithm (most likely some neural network) is not publicly available

[thomasdc]

According to the mentioned docs it uses cl100k_base.

[dmitry-brazhenko]

@thomasdc could you share please expected behavior?
Would you like to have smth like ?
var encoding = GptEncoding.GetEncodingForEmbeddingModel("text-embedding-3-large");

[thomasdc]

GptEncoding.GetEncodingForModel("text-embedding-ada-002"); was working fine, but throws for text-embedding-3-large

[dmitry-brazhenko]

https://github.com/dmitry-brazhenko/SharpToken/pulls

[dmitry-brazhenko]

Thanks for your feddback. I just introduced this change. Should work in the newer version (1.2.15) once it is available in nuget. Probably in 1-2 hours

[thomasdc]

Ok thanks, can you add small as well?

[dmitry-brazhenko]

@thomasdc could you share the full name please? I will add it

[dmitry-brazhenko]

Sorry, I missed it.
#29
it should become available in v.1.2.16
Please let me know if smth does not work