Comments (4)
For what it's worth, this behavior is documented. Idx::new
was designed to ensure that the returned index intendes to refer to an item on index, rather than null/void.
Panic
Panics if the ID is too large (>= 2^63)
One would use Idx::none
to create a void index (internally represented by -1
).
What one can do to improve this is:
- Extend the documentation of these functions to further clarify that
Idx::new
really expects a non-negative idx. - Provide a
new_unchecked
variant which admits any idx value.
from faiss-rs.
For what it's worth, this behavior is documented
My mistake I missed that
Idx::new was designed to ensure that the returned index intendes to refer to an item on index, rather than null/void.
I'm new to faiss so I could be misunderstanding something. Is it the case that faiss requires Idx to be non-negative? My use case is I'm adding vectors with add_with_ids
, and the ids I'm using are completely arbitrary and random; so when trying to add a vector with a given id I can generate a negative idx. The thing is the "void index" thing seems to be faiss-rs thing, as I can't find any indication that you can't use a negative idx with faiss. If faiss allows you to use negative indexes then isn't an Option
better here?
from faiss-rs.
Just catching up with older issues.
Is it the case that faiss requires Idx to be non-negative?
That much is the assumption made in this API, but it does not go far off from what is expected to users of the native Faiss API. An index entry ID of -1
in a search result means an empty/missing entry, which can happen in some index implementations.
My use case is I'm adding vectors with add_with_ids, and the ids I'm using are completely arbitrary and random; so when trying to add a vector with a given id I can generate a negative idx. The thing is the "void index" thing seems to be faiss-rs thing, as I can't find any indication that you can't use a negative idx with faiss.
As explained above, this concept of empty ID is indeed in the native library. Even if this API were to be more relaxed to allow negative numbers, it should not allow an ID value of -1
because that collides with the semantic of a missing vector on the search result list. You would not be able to distinguish a search result item from being empty or from being the vector of ID -1
. Sounds like a very nasty source of bugs to me.
All of this can be circumvented by converting an i64
into an Idx
via From<i64>
, but for this to be sound, you would need to adjust the ID generation logic in any case, so as to avoid generating -1
.
If faiss allows you to use negative indexes then isn't an Option better here?
It would be an interesting design option, but without an integer type that is not allowed to be -1
, that would bloat the size of the index type, making it terribly inefficient. Right now, idx_t
values are mapped 1:1 to Idx
without representation changes. Forcing us to output a vector of Option<Idx>
as the result would require a copy onto a representation that can hold the 64-bit integer plus the Option
discriminant.
from faiss-rs.
An index entry ID of -1 in a search result means an empty/missing entry, which can happen in some index implementations.
I missed this; this would complicate things
from faiss-rs.
Related Issues (20)
- Improve PreTransformIndex and update Clustering types HOT 1
- Macros to derive crate
- Support faiss_index_gpu_to_cpu in GpuIndexImpl HOT 2
- Use cxx for interop with Faiss HOT 1
- Slow training IVF indexes HOT 13
- Control over GPU <-> CPU result copies? HOT 1
- Memory leak detected. HOT 4
- Update C-API v1.7.2 HOT 1
- Save and reopen Index after train()? HOT 1
- read_index io flag support HOT 3
- Undefined symbols for architecture arm64
- Write Index into memory or Read Index from memory HOT 1
- Does this crate plan to support generic IO? HOT 1
- Can you also add support for static libraries? HOT 6
- Updating to latest faiss? HOT 9
- Support for mac m1? HOT 7
- Support binary indexes HOT 5
- confusing build linker error HOT 2
- `&mut self` argument on `index.search` HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from faiss-rs.