Brute Force vs hnsw Problem

[jelmerk]

Brute Force vs hnsw Problem

[jelmerk]

brute force does a pairwise comparison. so it will compare each embedding in the set with each other embedding.

On spark you can use KnnSimilarityEvaluator to evaluate the accuracy

Or do it by hand as you did

[ccalvin97]

Thanks for your reply. I am curious about, in general, how much recall can this model reach in small dataset like 1000-10000? thanks

[jelmerk]

depends on how similar items are. if the best match is like 0.5 and the worst 0.45 its not going to perform that well