hnswlib-spark slow on the retrieval of millions items

[jelmerk]

hnswlib-spark slow on the retrieval of millions items

[jelmerk]

It's hard to say without knowing your setup.

But some general tips

1. Use the suggested configuration from the readme (make sure speculation and dynamic allocation are off etc)
2. use inner product with a normalizer instead of cosine distance
3. increasing the number of partitions will speed up indexing at the expense of slowing down querying
4. increasing the number of replicas (setNumReplicas) will speed up querying at the expense of resource usage, you use if you set it to one, you will need to double the number of executors you use when querying

A good way to work out why things are slow is to look at the logs and look for lines that contain the word partition

[wangruinju]

Thanks for the tips!