- ๐ญ Co-founder at Databend Labs, a startup that focuses on building the world's best data cloud on top of the open-source project databend
- ๐ฏ Contributor to ClickHouse and TokuDB
- ๐ซ How to reach me: Blog (Chinese) | Twitter
- ๐ Pronouns: He/Him, BohuTANG (ๅผ ้้ฃ)
bohutang / tiktoken-rs Goto Github PK
View Code? Open in Web Editor NEWThis project forked from zurawiki/tiktoken-rs
Ready-made tokenizer library for working with GPT and tiktoken
License: MIT License