[Bug]: [cluster-mode] Milvus crash and cannot auto-recover successfully

[tianshihan818]

[Bug]: [cluster-mode] Milvus crash and cannot auto-recover successfully

[xiaofan-luan]

2. The minio disk space is enough, only 25% used now, so I am super wondering the reason. And referring to your suggestions, I am doing a new test and preparing to grab the milvus cluster & node metrics at that crash time, will update here later.

the reason is you are using IVF_SQ8. This index don't have original vectors.
Milvus try to prewarm original vectors on local cache in order to improve the cold latency.

if you don't need to get vectors, change queryNode.cache.warmup to off could help

[xiaofan-luan]

if you don't have enough space on /var, the K8s will automatically evict, that's might be the reason why you can not recover

[yanliang567]

/assign @tianshihan818
please try as comments above.

/unassign

[tianshihan818]

2. The minio disk space is enough, only 25% used now, so I am super wondering the reason. And referring to your suggestions, I am doing a new test and preparing to grab the milvus cluster & node metrics at that crash time, will update here later.

the reason is you are using IVF_SQ8. This index don't have original vectors. Milvus try to prewarm original vectors on local cache in order to improve the cold latency.

if you don't need to get vectors, change queryNode.cache.warmup to off could help

Thanks for your advice!
I have tried this, to set the queryNode.cache.warmup to be off. But it doesn't work, still crash with the same reason.
And I also tried to delete and rebuild HNSW index on this collection and reload, it works well and the directory /var disk usage is normal.
So I'm sure this problem is mainly due to the working line of the IVF_SQ8 index type, although I don't know the exactly reason.

[xiaofan-luan]

if you need to get original vector on search, ivfPQ can not do this. you need to specify a disk volume large enough for hosting all the raw vectors. otherwise even if you don't do warmup the size will keep growing and fail at some time

[tianshihan818]

if you need to get original vector on search, ivfPQ can not do this. you need to specify a disk volume large enough for hosting all the raw vectors. otherwise even if you don't do warmup the size will keep growing and fail at some time

I got it. I check the system configurations of milvus again, and find this: https://milvus.io/docs/system_configuration.md#localStorage
So the minio only stores the insert logs and the index files for data, and for the index like IVF_SQ or IVF_PQ, it will load all raw data during search/query, and the default path is /var/lib/milvus/data.
If I want to avoid that crash scene, I need to set localStorage.enabled to be true then change the localStorage.path to another place with enough disk space, right?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

[yanliang567]

@tianshihan818 did you try that? and how it works?

[tianshihan818]

@tianshihan818 did you try that? and how it works?

@yanliang567 Yes I tried but find that the helm chart config has hard-coded the path. Related issue: #33524
I think this issue could be closed as the key problem is clear.