instant-labs / instant-clip-tokenizer Goto Github PK
View Code? Open in Web Editor NEWLicense: MIT License
License: MIT License
Expose the functionality of the Rust crate to Python, by creating a wrapper library using PyO3.
In addition to the methods on Tokenizer
there should also be a convenience method that takes a list of strings, tokenizes all of them, adds <start_of_text>
and <end_of_text>
markers to each, truncates to a given context length, and writes the results in the rows of a numpy array. This corresponds to the tokenize
convenience function from the Python library.
Open questions:
decode
convenience function from the Python library? We do not use this anywhere, and it does not even work on my computer (out of the box).Right now, the Tokenizer::encode
method lowercases its input string. It is questionable whether this is the correct place to do this as it is some form of pre-processing, or whether this should be the responsibility of the caller (that is, one has to pass in lowercased text).
Option 1 (keep it in in Tokenizer::encode
):
Option 2 (let the caller do it):
Currently, the Tokenizer::with_vocabulary
constructor returns a std::io::Error
error, abusing the io::ErrorKind::Other
variant for situations where the vocabulary data format is invalid.
We should probably provide a custom crate-level Error
type and use that instead, with one variant for I/O-errors and further variants for invalid data formats.
Some simple CI/CD setup is necessary. At least the following should be checked:
These seem to be consistently wrapping at 80 columns instead of the 100 we mostly use (matching the code).
Instead of using a regular expression for splitting the input text into words (called Tokenizer::word_split
) we could write a custom parser, thereby getting rid of the dependency on the regex
crate, and also maybe improving performance slightly.
We should compare the output of the tokenization (and also decoding) of the Rust implementation against the original Python code on some large dataset, to ensure the Rust implementation is 100% compatible.
Questions:
It would be good to have some simple fuzz tests, since the inputs to the tokenizer will often be untrusted user input.
In addition to just checking for panics we could also verify in the fuzz test that lowercase_and_strip_whitespace(decode(encode(input))) == lowercase_and_strip_whitespace(input)
holds for every input
where lowercase_and_strip_whitespace
lowercases its input and removes all whitespace.
The original Python library uses an internal dictionary (SimpleTokenizer#cache
) where it stores the tokenization results for each word it has tokenized so far.
This obviously speeds things up for frequently occurring words, however, the drawback is that this cache is unbounded in size. Therefore, if the tokenizer is used to tokenize user-provided input (e.g. for a search API-endpoint) then a malicious user could feed the tokenizer with random sequences of characters, eventually resulting in an out-of-memory condition.
For this reason the Rust implementation does not yet use an internal cache.
We have a few options here:
AHashMap
but instead something bounded like a LRU-cache)trait Cache
and change the Tokenizer::encode
method to fn encode<C: Cache>(&self, text: &str, out: &mut Vec<Token>, cache: &mut C)
. We should probably also provide some simple implementations like NoCache
(doesn't do anything) and UnboundedCache
(just uses a AHashMap
- equivalent to Python version). If a user wants something more fancy than that they can implement it for themselves.A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.