Comments (4)
Just realized that there is one big blocker: we expose ndarray types in interfaces and ndarray is not 1 yet.
If we'd go to version 1 now, we would be stuck with ndarray 0.12.
from finalfusion-rust.
Spontaneously adding a chunk for word frequencies comes to mind, I believe we have discussed this somewhere in the past but I forgot what the outcome was.
That would require changes to Embeddings::new
which is part of the public API unless we'd want to expose some set_frequencies
method post 1.0.0.
Another thing that comes to mind is that we could discuss batched embeddings lookup, taking &[&str]
and returning Array2<f32>
, although I'm not sure if such a method would be too niche. This probably wouldn't touch any existing API, unless we'd want an abstraction for embedding()
too handle both cases.
Eliminating dead buckets would be another change where I'm unsure how it touches existing APIs. This would probably implemented by adding a new vocabulary type which in turn would probably add a new variant to the public VocabWrap
enum.
There might be more ideas lurking around, I'll add those here later.
from finalfusion-rust.
Judgin by rust-ndarray/ndarray#670 the next release will be 0.13 with no signs of a 1.x release any time soon.
from finalfusion-rust.
One other thing that I still want to do that also extends enums is the mmap'ed variant of quantized embedding matrices. I will put this high on my todo list, because it would be nice to have this in sticker.
from finalfusion-rust.
Related Issues (20)
- Test non-Linux UNIX platforms HOT 1
- NdNorms ChunkIdentifier not in header. HOT 1
- Quantized Array interface HOT 1
- Documentation outdated in 'io.rs'? HOT 1
- Zero-allocation lookups HOT 3
- Analogy interface is too inflexible
- Vocab should be a supertrait of SubwordVocab HOT 1
- Memory mappable vocab HOT 2
- How to use Ngrams to find words not in dictionary HOT 2
- compiling to wasm HOT 2
- Capacity overflow on Windows HOT 3
- Release Explicit Conversions? HOT 3
- Indexing a BucketVocab with the empty String returns Some HOT 1
- Support for phrase embeddings HOT 1
- Relicensing to MIT or Apache License version 2.0 HOT 3
- Reading GoogleNews word2vec model fails HOT 1
- Add support for word2vec keyed vector HOT 4
- Question: why don't we have support for computing analogy test accuracy for embeddings with quantized storage? HOT 1
- Release 0.11 HOT 25
- Properly handle memory mapped storage on big-endian platforms HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from finalfusion-rust.