holsee/spirit_fingers

Elixir SimHash NIFs written in Rust

SpiritFingers

"Fast SimHash NIFs written in Rust 🐇💨 as Erlang/Elixir versions were too slow 🐢"

For a full comparison versus native elixir solutions see: Simhash Benchmarks TL;DR Spirit Fingers is 400-900x faster, orders of magnitude more memory efficient and handles large binaries where others cannot.

• Hex Package.
• Documentation.

Build

mix compile

Test

mix test

Versions

• Elixir ~> 1.14
• OTP 25
• Rust 2021 ~> 1.67.0
• Rustler 0.27.0

Installation

Add spirit_fingers to your list of dependencies in mix.exs:

def deps do
  [
    {:spirit_fingers, "~> 0.4"}
  ]
end

Usage

Generate SimHash:

SpiritFingers.SimHash.similarity_hash("wow")
{:ok, 17399923637769257768}

Similarity between strings:

SpiritFingers.SimHash.similarity("Hocus pocus", "Hocus pocus pilatus pas")
{:ok, 0.9375}

Hamming Distance between hashes:

SpiritFingers.SimHash.hamming_distance(17399923637769257768, 17399923637769257768)
{:ok, 0.0}

Similarity between hashes:

SpiritFingers.SimHash.hash_similarity(17399923637769257768, 17399923637769257768)
{:ok, 1.0}

spirit_fingers's Issues

Fail to setup your lib

Hey mates, thanks for the awesome contribution to the Elixir community. I tried to add this package to my phoenix app and a little bit confused about why it not working. Maybe I did something wrong. If you find some time to help I will be very appreciated.

elixir -> 1.11.1
rust -> 1.47.0

in mix.exs deps {:spirit_fingers, "~> 0.3.0"}

Looks like compilation passed weel:

mix deps.compile spirit_fingers:

Compiling NIF crate :simhash (native/simhash)...
    Finished release [optimized] target(s) in 0.02s

but when I started iex -S mix I can't access SimHash module and any of it functions. Feel so stupid to ask this but what I missed?

SimHash.simhash("wow")
** (UndefinedFunctionError) function SimHash.simhash/1 is undefined (module SimHash is not available)
    SimHash.simhash("wow")

Compilation fails with Elixir 1.8 and OTP 21

SpiritFingers contains a NIF that has to be compiled after installation. I was trying to compile it but it fails with this message:

$ mix deps.compile
...
==> spirit_fingers
Compiling NIF crate :simhash (native/simhash)...
   Compiling erlang_nif-sys v0.6.3
   Compiling syn v0.11.11
error: failed to run custom build command for `erlang_nif-sys v0.6.3`
process didn't exit successfully: `/Users/leonid/Projects/ex_lsh/_build/dev/rustler_crates/simhash/release/build/erlang_nif-sys-bf350dda8055f7a2/build-script-build` (exit code: 101)
--- stdout
Unsupported Erlang version.

Is the erlang_nif-sys version up to date in the Cargo.toml?
Does 'cargo update' fix it?
If not please report at https://github.com/goertzenator/erlang_nif-sys.

--- stderr
thread 'main' panicked at 'gen_api.erl encountered an error.', /Users/leonid/.cargo/registry/src/github.com-1ecc6299db9ec823/erlang_nif-sys-0.6.3/build.rs:28:13
note: Run with `RUST_BACKTRACE=1` for a backtrace.

After looking up on Rustler, I figured out that the NIF version I need is 2.14, so I tried this

RUSTLER_NIF_VERSION=2.14 mix deps.compile

but it fails like this:

==> spirit_fingers
Compiling NIF crate :simhash (native/simhash)...
   Compiling unicode-xid v0.0.4
   Compiling void v1.0.2
   Compiling erlang_nif-sys v0.6.3
   Compiling quote v0.3.15
   Compiling unicode-segmentation v1.2.1
   Compiling siphasher v0.2.2
   Compiling lazy_static v0.1.16
   Compiling lazy_static v0.2.11
   Compiling unreachable v0.1.1
   Compiling synom v0.11.3
   Compiling simhash v0.2.0 (https://github.com/holsee/simhash-rs.git#7fc44d12)
   Compiling heck v0.3.0
error: failed to run custom build command for `erlang_nif-sys v0.6.3`
process didn't exit successfully: `/Users/leonid/Projects/ex_lsh/_build/dev/rustler_crates/simhash/release/build/erlang_nif-sys-bf350dda8055f7a2/build-script-build` (exit code: 101)
--- stdout
Unsupported Erlang version.

Is the erlang_nif-sys version up to date in the Cargo.toml?
Does 'cargo update' fix it?
If not please report at https://github.com/goertzenator/erlang_nif-sys.

--- stderr
thread 'main' panicked at 'gen_api.erl encountered an error.', /Users/leonid/.cargo/registry/src/github.com-1ecc6299db9ec823/erlang_nif-sys-0.6.3/build.rs:28:13
note: Run with `RUST_BACKTRACE=1` for a backtrace.

warning: build failed, waiting for other jobs to finish...
error: build failed

What does SpiritFinger require to build out-of-the-box?

• Erlang version?
• Elixir version?
• Rust version?