Giter Site home page Giter Site logo

faiss-rs's Introduction

Faiss-rs

faiss at crates.io Continuous integration status Minimum Rust Version Stable dependency status

This project provides Rust bindings to Faiss, the state-of-the-art vector search and clustering library.

Installing with dynamic linking

By default, this crate is dynamically linked with the Faiss library installed in your system, so it does not build Faiss automatically for you. To build the library yourself:

  1. Follow the instructions here to build Faiss using CMake, enabling the variables FAISS_ENABLE_C_API and BUILD_SHARED_LIBS. The crate is currently only compatible with version v1.7.2. Consider building Faiss from this fork, c_api_head branch, which will contain the latest supported bindings to the C interface. For example:

    cmake -B build -DFAISS_ENABLE_C_API=ON -DBUILD_SHARED_LIBS=ON -DCMAKE_BUILD_TYPE=Release
    cmake --build build

    This will result in the dynamic library faiss_c ("c_api/libfaiss_c.so" on Linux), which needs to be installed in a place where your system will pick up (in Linux, try somewhere in the LD_LIBRARY_PATH environment variable, such as "/usr/lib", or try adding a new path to this variable. For mac users, this means placing it in /usr/local/lib/libfaiss_c.dylib). For GPU support, don't forget to enable the option FAISS_ENABLE_GPU. Note: faiss_c might link dynamically to the native faiss library, which in that case you will need to install the main shared object (faiss/libfaiss.so) as well.

  2. You are now ready to include this crate as a dependency:

    [dependencies]
    "faiss" = "0.11.0"

If you have built Faiss with GPU support, you can include the "gpu" Cargo feature:

[dependencies]
"faiss" = { version = "0.11.0", features = ["gpu"] }

Installing with static linking

Alternatively to the above, enable the "static" Cargo feature to let Rust build Faiss for you. You will still need the dependencies required to build and run Faiss as described in their INSTALL.md, namely a compatible C++ compiler and a BLAS implementation.

[dependencies]
"faiss" = { version = "0.11.0", features = ["static"] }

Compiling Faiss with GPU support is also possible.

[dependencies]
"faiss" = { version = "0.11.0", features = ["static", "gpu"] }

Using

A basic example is seen below. Please check out the documentation for more.

use faiss::{Index, index_factory, MetricType};

let mut index = index_factory(64, "Flat", MetricType::L2)?;
index.add(&my_data)?;

let result = index.search(&my_query, 5)?;
for (i, (l, d)) in result.labels.iter()
    .zip(result.distances.iter())
    .enumerate()
{
    println!("#{}: {} (D={})", i + 1, *l, *d);
}

License and attribution notice

Licensed under either of

at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

This work is not affiliated with Facebook AI Research or the main Faiss software.

faiss-rs's People

Contributors

agerasev avatar ava57r avatar enet4 avatar gensmusic avatar mooreniemi avatar skyfan2002 avatar tronicboy1 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

faiss-rs's Issues

Convert IndexImpl to IdMap<IndexImpl>

Hello.

Example

let index = FlatIndexImpl::new_l2(4).unwrap();
let id_map = IdMap::new(index).unwrap();
write_index(&id_map, "/tmp/id_map_ex.dat").unwrap();
let new_index = read_index("/tmp/id_map_ex.dat").unwrap();

new_index is IndexImpl

`&mut self` argument on `index.search`

Index.search takes a &mut self argument. Is it safe to change this to &self? I had assumed that .search didn't mutate anything, and not being able to concurrently do searches is somewhat problematic for my application.

confusing build linker error

Hopefully this is a simple solution to something basic I might have missed.

Trying to utilize the static feature to compile a project that consumes faiss-rs as a crate.

Example Cargo.toml

[dependencies]
...
faiss = { version = "0.12.1", features = ["static"] }
  • Run cargo build
   Compiling json v0.12.4
   Compiling jni v0.21.1
   Compiling faiss v0.12.1
error: could not find native static library `faiss`, perhaps an -L flag is missing?

error: could not compile `faiss-sys` (lib) due to 1 previous error
warning: build failed, waiting for other jobs to finish...

I'm not super experienced here but, isn't the static library faiss what the static feature is supposed to build and provide in the first place? Am I missing something basic here? Sorry if this is a dumb question.

"Linking with cc failed" -- issue compiling?

Hey folks! I'm having what appears to be some issue compiling. Do you have any thoughts or suggestions about what might be going on here?

The closest I can find seems to point to a cargo or mac error:
rust-lang/rust#25289 (comment)
rust-lang/rust#60149 (comment)

I'm really quite a newbie at rust, so any debugging advice is appreciated!

Code :

use faiss::{index_factory, MetricType};

pub fn run_query() -> Option<f32>{
    let mut index = index_factory(3, "Flat", MetricType::L2).unwrap();
    Some(0.0)
}

Dependency:

[dependencies]
"faiss" = {version = "0.6.0"}

Error:

error: linking with `cc` failed: exit status: 1
  |
  = note: "cc" "-m64" "-arch" "x86_64" "-L" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/thistle.1npak595a7css8z4.rcgu.o" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/thistle.20hce9hmqhl203kn.rcgu.o" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/thistle.2m6hiy95l4iyrwu5.rcgu.o" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/thistle.2wd4t96zh75gv0va.rcgu.o" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/thistle.3aclf5rjqxhrola4.rcgu.o" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/thistle.chnosn5cykoslmi.rcgu.o" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/thistle.sf29mzngf3nyfuf.rcgu.o" "-o" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/thistle" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/thistle.1vtd5t81sh40jhul.rcgu.o" "-Wl,-dead_strip" "-nodefaultlibs" "-L" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps" "-L" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/build/torch-sys-ff3e9973ea28b04e/out/libtorch/libtorch/lib" "-L" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/build/torch-sys-ff3e9973ea28b04e/out" "-L" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/build/bzip2-sys-7fb991f7dbc50a4c/out/lib" "-L" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libthistle-d4a6c2bff82dd787.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librust_bert-31fa07c451540f80.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libitertools-1dfefb650b8cd53d.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librust_tokenizers-c31d0d587cde9540.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libcsv-ed679ac6734a6098.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libcsv_core-d75bfe6d0b9a8acb.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libbstr-c0070e4a1cc2d79c.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libregex_automata-87878d88ec463ae2.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libserde_json-21d54dcd579fec67.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libryu-90cf68ddeda11aee.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libitoa-abc241418b49187a.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libunicode_normalization-b2ce0f79e7c9edf9.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libtinyvec-1c50f201c52e47ed.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libtinyvec_macros-58d800761f45510a.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libregex-7a5e938964f73b44.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libregex_syntax-5efabd26812b2a98.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libaho_corasick-35270916c15955fe.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libmemchr-3f7b52af37b6f9bd.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libitertools-ae0eacccf85a2ec4.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librayon-bc7a015cc78390f5.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librayon_core-99e1dad4e52206ec.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libnum_cpus-0cc61c5a851016e3.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libcrossbeam_deque-8134651f99d6d55c.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libcrossbeam_epoch-5f4b888d78829961.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libmemoffset-6a6eb427edffc44b.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libscopeguard-31f20504eb2703cc.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libcrossbeam_channel-6756a1e59e54d8c6.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libcrossbeam_utils-a78a168a13cd1267.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libeither-562e7f0a923e8f49.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libordered_float-51fc742f3bcfcf51.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libtch-574ba415f0a7b978.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libndarray-20eeeae1632d9fd8.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libmatrixmultiply-f4c1da960d45074e.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libnum_complex-d55c58eb6e91e88f.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libnum_integer-e553cc3e4c6712d2.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libnum_traits-a75e93e9dc909b7f.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librawpointer-af04cc4bb8e47631.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libtorch_sys-2d9d21974de55caa.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libzip-c44b3f38577791de.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libtime-b0c05a3d813ace7b.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libthiserror-68a9b42d4132521f.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libbzip2-896046b3e15dd93a.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libbzip2_sys-e17a3ea19e42b8cc.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libflate2-6e853baf9c8b9e50.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libminiz_oxide-8509711269a1300e.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libadler32-51f964c75e3afbf2.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libcfg_if-892e7cee77ae3329.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libbyteorder-be28ddbc530eab12.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libcrc32fast-6110fcd70cf2e9da.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libfailure-ae0aa4a153df8b9f.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libbacktrace-093e0afaa529e2b3.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libobject-c15f758dd1367b9c.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libaddr2line-1fc0fb83f501e321.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libgimli-6174ef008c83096e.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libcfg_if-a42ba01468c6d92e.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librustc_demangle-9cc4760bb3afbdef.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/liblazy_static-98ea06d08472431a.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libfaiss-c6a410c6f8ab2423.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libfaiss_sys-74aeabac60e0cbd0.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libuuid-7e702774b0c69b2c.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/libserde-7d7b34b7398292db.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librand-90af94b665a55644.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librand_xorshift-71f5c0bf8db7f494.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librand_pcg-c9f195549219f059.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librand_hc-7600fcdf91228ccb.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librand_chacha-f72b2299b81d26aa.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librand_isaac-8e1f8597c5ec4d3f.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librand_core-4170c848b672f583.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librand_os-5e84c8953b0ba2d6.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librand_jitter-3cd042c23570d8ca.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/liblibc-c2fdc019e3d38deb.rlib" "/Users/bradwindsor/ms_projects/thistle/thistle/target/debug/deps/librand_core-92e42927a2b53f93.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libstd-46a061b08cc06738.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libpanic_unwind-7bebd2f04d2b9056.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libobject-8022bf77ecaec932.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libaddr2line-40998048b1ba428e.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libgimli-bd2a545ac887ab20.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libstd_detect-c0832a78f281aa17.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/librustc_demangle-e84927fb7a338bcb.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libhashbrown-80e6dd310746668f.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_alloc-70160232b7f5a1eb.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libunwind-dc1f6511b0655580.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcfg_if-7606550a486f0fa7.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/liblibc-d71be2efb2f94c3f.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/liballoc-6bbd477ed289203f.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/librustc_std_workspace_core-c7c7638eea6d267e.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcore-ac475f5c856023a5.rlib" "/Users/bradwindsor/.rustup/toolchains/nightly-x86_64-apple-darwin/lib/rustlib/x86_64-apple-darwin/lib/libcompiler_builtins-35393beafda280dd.rlib" "-lc++" "-ltorch" "-ltorch_cpu" "-lc10" "-lfaiss_c" "-framework" "Security" "-liconv" "-lSystem" "-lresolv" "-lc" "-lm" "-liconv"
  = note: ld: library not found for -lfaiss_c

Updating to latest faiss?

I've forked @SkyFan2002 's static build, and in the effort to simplify, I've also updated to latest Faiss:
https://github.com/jondot/faiss-rs
Compiling on Mac M1 and running vector search and indexing works well.

As now we have static build in this repo, I'm wondering if its possible to point to a recent Faiss version (and not a specific C_api flavor), or there's something hidden from my knowledge that forbids that?

Support for mac m1?

I successfully installed faiss via brew with command:
brew reinstall faiss --build-from-source
it runs
cmake -B build . -DFAISS_ENABLE_GPU=OFF -DFAISS_ENABLE_PYTHON=OFF -DFAISS_ENABLE_C_API=ON -DBUILD_SHARED_LIBS=ON.
I could add to my project faiss = "0.12.1", but when I try to run it it says note: ld: library not found for -lfaiss_c

Write Index into memory or Read Index from memory

Hello.

C++ Faiss supports some methods for saving a index into memory.

write into IOWriter
https://github.com/facebookresearch/faiss/blob/main/faiss/index_io.h#L40
and impl VectorIOWriter https://github.com/facebookresearch/faiss/blob/main/faiss/impl/io.h#L59

read from IOReader
https://github.com/facebookresearch/faiss/blob/main/faiss/index_io.h#L61
and impl VectorIOReader
https://github.com/facebookresearch/faiss/blob/main/faiss/impl/io.h#L53

Could We add these impls into rust library?

panicked at 'too large index value provided to Idx::new'

When you use a large integer (greater than 2^63) as an ID, faiss-rs will panic. This is a bit unexpected as it's not mentioned anywhere in the documentation, nor do I think faiss has this limitation. If the intent is to represent null ids, isn't Option<faiss::idx_t> a better choice?

Slow training IVF indexes

I tried comparison two kinds of program python and c_api

Both test lib use OpenBlas and LAPACK.

OS: Oracle Linux 8.3

Python results

faiss lib from anaconda channel conda-forge

In python on GPU

import numpy as np
import faiss

d = 128
res = faiss.StandardGpuResources()

index = faiss.index_factory(d, "IVF16384,Flat")
index.verbose = True
select_gpu = 0

xt = faiss.rand((1000000, d))
index_ivf = faiss.extract_index_ivf(index)
clustering_index = faiss.index_cpu_to_gpu(res, select_gpu, faiss.IndexFlatL2(d))
index_ivf.clustering_index = clustering_index

index.train(xt)
time python3 train-gpu-rand.py 
Training level-1 quantizer
Training level-1 quantizer on 1000000 vectors in 128D
Training IVF residual
IndexIVF: no residual training

real    0m8.057s
user    1m59.744s
sys     0m17.446s

In python on CPU

import numpy as np
import faiss

d = 128
index = faiss.index_factory(d, "IVF16834,Flat")
index.verbose = True

xt = faiss.rand((1000000, d))
index.train(xt)
time python3 train-cpu-rand.py 
Training level-1 quantizer
Training level-1 quantizer on 1000000 vectors in 128D
Training IVF residual
IndexIVF: no residual training

real    7m13.904s
user    403m29.746s
sys     519m31.349s

Rust bindings

in Rust on CPU
use faiss-rs v0.10

use rand::prelude::*;
use faiss::Index;

fn main() {
    let d:usize = 128;
    let mut rng = rand::thread_rng();
    let mut xt:Vec<f32> = Vec::with_capacity(1_000_000 * d);
    for _ in 0..(1_000_000 * d) {
        xt.push(rng.gen());
    }

    let mut index = faiss::index_factory(
        d as u32,
        "IVF16834,Flat",
        faiss::MetricType::L2,
    )
    .unwrap();

    index.set_verbose(true);
    index.train(&xt).unwrap();
}

build from source faiss lib
facebookresearch/faiss#1838
from branch https://github.com/ava57r/faiss/tree/c_api_avx2 commit ava57r/faiss@02ecc5b

options

cmake -B build \
-DBUILD_TESTING=OFF \
-DFAISS_ENABLE_GPU=OFF \
-DFAISS_OPT_LEVEL=avx2 \
-DFAISS_ENABLE_C_API=ON \
-DFAISS_ENABLE_PYTHON=OFF \
-DBUILD_SHARED_LIBS=ON \
-DCMAKE_BUILD_TYPE=Release .
make -C build -j

results
I kill the process

first time

   Compiling train-faiss-cpu v0.1.0 (/home/faiss-dev/train-faiss-cpu)
    Finished release [optimized] target(s) in 0.36s
     Running `target/release/train-faiss-cpu`
Training level-1 quantizer
Training level-1 quantizer on 1000000 vectors in 128D
^C

real    70m43.693s
user    1032m34.066s
sys     5m17.008s

second time

     Running `target/release/train-faiss-cpu`
Training level-1 quantizer
Training level-1 quantizer on 1000000 vectors in 128D
^C

real    16m37.219s
user    242m21.940s
sys     1m22.869s

I can't find a reason this behavior.

Could We need to add benckmark(s)?

Memory leak detected.

Hello,

I use the crate-0.10.0 for our embedding recall, and have found some tips:
a. IndexImpl has not impl concurrentIndex, so IVFScalarQuantizerIndexImpl returned by into_ivf_scalar_quantizer builds error;
b. search increase the leaking of memory without upbound limit, so does the read_index, dropped but no release the RES memory.

What would the problem be? Anybody encount?

Thanks

Faiss c_api dynamic library is called libfaiss_c.so not faiss_c

I've just worked through the readme to get this to build and the output of make-ing faiss's c_api is called libfaiss_c.so.

Copying this to /usr/lib and changing faiss-sys/Cargo.toml allows me to build this project again.

Happy to put a PR in to fix this up, I just wanted to flag this up prior to that incase it was a mistake on my part in the setup.

Support binary indexes

Hi there!

Thanks for maintaining these bindings!

So, iiuc faiss has had support for binary indexes since v1.6.2 and faiss-rs is built against v1.7.2 so I naturally assumed I'd have faiss::read_index_binary, etc methods available but that doesn't seem to be the case.

I'm not very well-versed with bindings and the whole C++-Rust interop so correct me if I'm wrong but I believe we're currently just not generating bindings for the binary indexes.

If getting this to work might not be a tall order, then would you be interested in guiding me so that I can contribute?

Thanks!

use cpp interface instead of c API

Having played with accessing faiss via the cxx crate it was reasonably straightforward to create interoperability between rust and faiss.

Is there a reason why the use of the C api is preferred for interacting with faiss?

Undefined symbols for architecture arm64

I got error on my macbook m1.

error: linking with `cc` failed: exit status: 1
  |
  = note: "cc" "-arch" "arm64" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.15y0mxaisygeybv4.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.16iynio6ure42lyx.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.1c0oi8mrf0degq2t.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.1ner4ngc0suidgfd.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.1r95aynkrsipxt9r.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.1vv4ed0ctn2rvdgx.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.2exg089oh2acxtky.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.2jydhusom7z0qvrb.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.2nxg59rqggu3e9bm.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.2op2z2qcimz5jqo6.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.2utg5l6dwbejz0la.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.359gjbd38ok5ohpz.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.3fo5oeaadvbggb3z.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.3ig2w098c8pg1k72.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.42m8dwm7d1dwehkd.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.4ldb0bo6cgv8aedq.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.4qlttl83bb0ye7gv.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.4vdk8serce0swo7t.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.50p0on0z6dv0lfq3.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.52nouuc1sh0dvjzc.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.54l75eewhw4t7dy.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.ky72hj9gysvajjv.rcgu.o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0.17gubvxp0bq6hj0e.rcgu.o" "-L" "/Users/kingzcheung/rust/tris/target/debug/deps" "-L" "/opt/homebrew/Cellar/faiss/1.7.2/lib" "-L" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib" "/Users/kingzcheung/rust/tris/target/debug/deps/libfaiss-18908e24c0be1756.rlib" "/Users/kingzcheung/rust/tris/target/debug/deps/libfaiss_sys-103c102dfb14c0d0.rlib" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libstd-0dd069df8d91d43f.rlib" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libpanic_unwind-7cbaf788dfce5d66.rlib" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libobject-af5296f79b444c55.rlib" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libmemchr-f15ae47b27e6c368.rlib" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libaddr2line-3e9ec6def1f5cffe.rlib" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libgimli-302f1e935cf05b0d.rlib" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/librustc_demangle-bb4e9f007ffeee2c.rlib" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libstd_detect-c9d4bbaea0812982.rlib" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libhashbrown-69fbf5632ee1531d.rlib" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libminiz_oxide-d0d3793083fe7a23.rlib" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libadler-e9688be92d7fdfea.rlib" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/librustc_std_workspace_alloc-1a07d2235b647a42.rlib" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libunwind-f4705be43877013e.rlib" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libcfg_if-218bb6979b4b9957.rlib" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/liblibc-5f8d5e70b747a13a.rlib" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/liballoc-8a2a692899862816.rlib" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/librustc_std_workspace_core-b8205ba56f89023d.rlib" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libcore-76ce7e0c1f6d7b3f.rlib" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libcompiler_builtins-4014697324d67e93.rlib" "-lfaiss_c" "-lSystem" "-lresolv" "-lc" "-lm" "-liconv" "-L" "/Users/kingzcheung/.rustup/toolchains/stable-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib" "-o" "/Users/kingzcheung/rust/tris/target/debug/deps/faiss_demo-3d8cabe14327e0d0" "-Wl,-dead_strip" "-nodefaultlibs"
  = note: Undefined symbols for architecture arm64:
            "_faiss_Index_is_trained", referenced from:
                _$LT$faiss..index..IndexImpl$u20$as$u20$faiss..index..Index$GT$::is_trained::hb658f6bf133d59b5 in libfaiss-18908e24c0be1756.rlib(faiss-18908e24c0be1756.faiss.5cda5eee-cgu.2.rcgu.o)
            "_faiss_get_last_error", referenced from:
                faiss::error::NativeError::from_last_error::h8c9f45b5d85d2050 in libfaiss-18908e24c0be1756.rlib(faiss-18908e24c0be1756.faiss.5cda5eee-cgu.7.rcgu.o)
            "_faiss_Index_free", referenced from:
                _$LT$faiss..index..IndexImpl$u20$as$u20$core..ops..drop..Drop$GT$::drop::heaac5513632449a0 in libfaiss-18908e24c0be1756.rlib(faiss-18908e24c0be1756.faiss.5cda5eee-cgu.2.rcgu.o)
            "_faiss_index_factory", referenced from:
                faiss::index::index_factory::hbcd8721430e299d8 in faiss_demo-3d8cabe14327e0d0.42m8dwm7d1dwehkd.rcgu.o
          ld: symbol(s) not found for architecture arm64
          clang: error: linker command failed with exit code 1 (use -v to see invocation)
 

Save and reopen Index after train()?

Looking at Index it's not obvious to me if there's a way I can save the index after it's been trained to avoid the cost of training it again on reload.

Would I just use inner_ptr and from_inner_ptr (but where would I get the size of the index in memory behind that ptr)?

Would you consider something like a save() method PR?

CI is halted, migrate to GitHub Actions

With Travis CI limiting free use in public projects, CI stopped running for this repository. An equivalent pipeline needs to be written using GitHub Actions.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.