[Bug]: Getting error "index not supported" for GPU CAGRA index, please help

[kdabbir]

[Bug]: Getting error "index not supported" for GPU CAGRA index, please help

[yanliang567]

/assign @Presburger
/unassign

[kdabbir]

@yanliang567 @Presburger is there another milvus version/release that successfully works with GPU_CAGRA? I think this is a regression seeing another ticket for DiskANN: #34222, We are ok to test with any version as long as it works

[xiaocai2333]

@kdabbir I think you didn't enable GPU when building the image; by default, it's a CPU image. Please check it.

[Presburger]

@kdabbir Hi, could you provide more detailed log information?

[kdabbir]

Attaching screenshot of GPU image.
Tried to get server version using below code:

print(fmt.format("start connecting to Milvus"))
connections.connect(host='localhost', port='19530')

has = utility.has_collection("hello_milvus")
print(f"Does collection hello_milvus exist in Milvus: {has}")


print("Checking milvus gpu version")
print(utility.get_server_version())

[kdabbir]

@kdabbir I think you didn't enable GPU when building the image; by default, it's a CPU image. Please check it.

All I did is docker pull milvusdb/milvus:v2.4.5-gpu and used this image

Hi, could you provide more detailed log information?

Sure, let me take an export of logs and attach

[kdabbir]

milvus-log.tar.gz
Attached logs (You'll need to scroll up in each log to get milvus pod log file since we have multiple metric container logs as well)

[kdabbir]

@xiaocai2333 @Presburger

[kdabbir]

logs taken from above zip in proxy and data coord:

collection=hello_milvus]
[2024/06/27 04:43:53.409 +00:00] [INFO] [proxy/impl.go:1996] ["CreateIndex received"] [traceID=8573b31b96cc174537765c7c57fbc273] [role=proxy] [db=default] [collection=hello_milvus] [field=embeddings] [extra_params="[{\"key\":\"metric_type\",\"value\":\"L2\"},{\"key\":\"index_type\",\"value\":\"GPU_CAGRA\"},{\"key\":\"params\",\"value\":\"{\\\"intermediate_graph_degree\\\":64,\\\"graph_degree\\\":32}\"}]"]
[2024/06/27 04:43:53.409 +00:00] [INFO] [proxy/impl.go:2009] ["CreateIndex enqueued"] [traceID=8573b31b96cc174537765c7c57fbc273] [role=proxy] [db=default] [collection=hello_milvus] [field=embeddings] [extra_params="[{\"key\":\"metric_type\",\"value\":\"L2\"},{\"key\":\"index_type\",\"value\":\"GPU_CAGRA\"},{\"key\":\"params\",\"value\":\"{\\\"intermediate_graph_degree\\\":64,\\\"graph_degree\\\":32}\"}]"] [BeginTs=450747022282326017] [EndTs=450747022282326017]
[2024/06/27 04:43:53.409 +00:00] [INFO] [proxy/task_index.go:438] ["proxy create index"] [traceID=8573b31b96cc174537765c7c57fbc273] [collectionID=450733024940556810] [fieldID=102] [indexName=] [typeParams="[{\"key\":\"dim\",\"value\":\"8\"}]"] [indexParams="[{\"key\":\"metric_type\",\"value\":\"L2\"},{\"key\":\"index_type\",\"value\":\"GPU_CAGRA\"},{\"key\":\"params\",\"value\":\"{\\\"intermediate_graph_degree\\\":64,\\\"graph_degree\\\":32}\"}]"] [newExtraParams="[{\"key\":\"metric_type\",\"value\":\"L2\"},{\"key\":\"index_type\",\"value\":\"GPU_CAGRA\"},{\"key\":\"params\",\"value\":\"{\\\"intermediate_graph_degree\\\":64,\\\"graph_degree\\\":32}\"}]"]
[2024/06/27 04:43:53.467 +00:00] [INFO] [proxy/impl.go:2027] ["CreateIndex done"] [traceID=8573b31b96cc174537765c7c57fbc273] [role=proxy] [db=default] [collection=hello_milvus] [field=embeddings] [extra_params="[{\"key\":\"metric_type\",\"value\":\"L2\"},{\"key\":\"index_type\",\"value\":\"GPU_CAGRA\"},{\"key\":\"params\",\"value\":\"{\\\"intermediate_graph_degree\\\":64,\\\"graph_degree\\\":32}\"}]"] [BeginTs=450747022282326017] [EndTs=450747022282326017]
[2024/06/27 04:43:53.754 +00:00] [INFO] [proxy/impl.go:6009] ["AllocTimestamp request receive"]
[2024/06/27 04:43:53.754 +00:00] [INFO] [proxy/impl.go:6018] ["AllocTimestamp request success"] [timestamp=450747022374076418]
[2024/06/27 04:43:54.546 +00:00] [DEBUG] [proxy/impl.go:2140] ["DescribeIndex received"] [traceID=9f259891fd34c3579a26214baa1f488b] [role=proxy] [db=default] [collection=hello_milvus] [field=] ["index name"=]
[2024/06/27 04:43:54.547 +00:00] [DEBUG] [proxy/impl.go:2155] ["DescribeIndex enqueued"] [traceID=9f259891fd34c3579a26214baa1f488b] [role=proxy] [db=default] [collection=hello_milvus] [field=] ["index name"=] [BeginTs=450747022583791618] [EndTs=450747022583791618]
[2024/06/27 04:43:54.548 +00:00] [DEBUG] [proxy/impl.go:2175] ["DescribeIndex done"] [traceID=9f259891fd34c3579a26214baa1f488b] [role=proxy] [db=default] [collection=hello_milvus] [field=] ["index name"=] [BeginTs=450747022583791618] [EndTs=450747022583791618]


[2024/06/27 02:52:57.828 +00:00] [DEBUG] [datacoord/import_scheduler.go:167] ["peek slots done"] [nodeSlots="{\"27\":16}"]
[2024/06/27 02:52:58.091 +00:00] [INFO] [datacoord/index_service.go:682] ["receive DescribeIndex request"] [traceID=6637014b369f196b4c0a9fc7f7fae072] [collectionID=450731453867370391] [indexName=] [timestamp=0]
[2024/06/27 02:52:58.091 +00:00] [INFO] [datacoord/index_service.go:603] ["completeIndexInfo success"] [collectionID=450731453867370391] [indexID=450731453867370404] [totalRows=33396] [indexRows=0] [pendingIndexRows=33396] [state=Failed] [failReason="450731453867570628: failed to create index, C Runtime Exception: index not supported\n: segcore unsupported error[segcoreCode=2003];"]
[2024/06/27 02:52:58.091 +00:00] [INFO] [datacoord/index_service.go:730] ["DescribeIndex success"] [traceID=6637014b369f196b4c0a9fc7f7fae072] [collectionID=450731453867370391] [indexName=]
[2024/06/27 02:52:58.091 +00:00] [INFO] [datacoord/index_service.go:682] ["receive DescribeIndex request"] [traceID=6637014b369f196b4c0a9fc7f7fae072] [collectionID=450733024940556810] [indexName=] [timestamp=0]
[2024/06/27 02:52:58.091 +00:00] [INFO] [datacoord/index_service.go:603] ["completeIndexInfo success"] [collectionID=450733024940556810] [indexID=450733024940556861] [totalRows=9003] [indexRows=6002] [pendingIndexRows=3001] [state=Failed] [failReason="450733024940556832: failed to create index, C Runtime Exception: index not supported\n: segcore unsupported error[segcoreCode=2003];"]
[2024/06/27 02:52:58.091 +00:00] [INFO] [datacoord/index_service.go:730] ["DescribeIndex success"] [traceID=6637014b369f196b4c0a9fc7f7fae072] [collectionID=450733024940556810] [indexName=]
[2024/06/27 02:52:58.176 +00:00] [DEBUG] [datacoord/services.go:660] ["DataCoord current state"] [StateCode=Healthy]
[2024/06/27 02:52:59.827 +00:00] [DEBUG] [datacoord/import_scheduler.go:167] ["peek slots done"] [nodeSlots="{\"27\":16}"]
[2024/06/27 02:53:01.090 +00:00] [INFO] [datacoord/index_service.go:682] ["receive DescribeIndex request"] [traceID=3a19b0373547be5a6d5e420fcf173ddf] [collectionID=450733024940556810] [indexName=] [timestamp=0]
[2024/06/27 02:53:01.090 +00:00] [INFO] [datacoord/index_service.go:603] ["completeIndexInfo success"] [collectionID=450733024940556810] [indexID=450733024940556861] [totalRows=9003] [indexRows=6002] [pendingIndexRows=3001] [state=Failed] [failReason="450733024940556832: failed to create index, C Runtime Exception: index not supported\n: segcore unsupported error[segcoreCode=2003];"]
[2024/06/27 02:53:01.090 +00:00] [INFO] [datacoord/index_service.go:730] ["DescribeIndex success"] [traceID=3a19b0373547be5a6d5e420fcf173ddf] [collectionID=450733024940556810] [indexName=]
[2024/06/27 02:53:01.090 +00:00] [INFO] [datacoord/index_service.go:682] ["receive DescribeIndex request"] [traceID=3a19b0373547be5a6d5e420fcf173ddf] [collectionID=450731453867370391] [indexName=] [timestamp=0]
[2024/06/27 02:53:01.090 +00:00] [INFO] [datacoord/index_service.go:603] ["completeIndexInfo success"] [collectionID=450731453867370391] [indexID=450731453867370404] [totalRows=33396] [indexRows=0] [pendingIndexRows=33396] [state=Failed] [failReason="450731453867570628: failed to create index, C Runtime Exception: index not supported\n: segcore unsupported error[segcoreCode=2003];"]
[2024/06/27 02:53:01.090 +00:00] [INFO] [datacoord/index_service.go:730] ["DescribeIndex success"] [traceID=3a19b0373547be5a6d5e420fcf173ddf] [collectionID=450731453867370391] [indexName=]
[2024/06/27 02:53:01.827 +00:00] [DEBUG] [datacoord/import_scheduler.go:167] ["peek slots done"] [nodeSlots="{\"27\":16}"]
[2024/06/27 02:53:02.953 +00:00] [DEBUG] [config/etcd_source.go:154] ["etcd refreshConfigurations"] [prefix=by-dev/config] [endpoints="[etcd-cagra-milvuspoctest2:2379]"]
[2024/06/27 02:53:03.714 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=b0d71425290cf428df9436e227173707] [collectionID=450731453867370391]
[2024/06/27 02:53:03.715 +00:00] [INFO] [datacoord/index_service.go:924] ["List index success"] [traceID=9acf55ea97cb2c9824b85fa94a410748] [collectionID=450731453867370391]
[2024/06/27 02:53:03.715 +00:00] [DEBUG] [datacoord/index_service.go:892] ["GetIndexInfos successfully"] [traceID=58db72095e502624cf2c92a323c6fbbc] [collectionID=450731453867370391] [indexName=]
[2024/06/27 02:53:03.717 +00:00] [INFO] [datacoord/services.go:820] ["get recovery info request received"] [traceID=18d08747b4b7f0fcaa639de7b907b31b] [collectionID=450731453867370391] [partitionIDs="[]"]
[2024/06/27 02:53:03.717 +00:00] [INFO] [datacoord/handler.go:117] [GetQueryVChanPositions] [collectionID=450731453867370391] [channel=by-dev-cagra-milvuspoctest2-dml_0_450731453867370391v0] [numOfSegments=1] ["indexed segment"=0]
[2024/06/27 02:53:03.717 +00:00] [INFO] [datacoord/handler.go:302] ["channel seek position set from channel checkpoint meta"] [channel=by-dev-cagra-milvuspoctest2-dml_0_450731453867370391v0] [posTs=450745265209737217] [posTime=2024/06/27 02:52:10.689 +00:00]
[2024/06/27 02:53:03.717 +00:00] [INFO] [datacoord/services.go:835] ["datacoord append channelInfo in GetRecoveryInfo"] [traceID=18d08747b4b7f0fcaa639de7b907b31b] [collectionID=450731453867370391] [partitionIDs="[]"] [channel=by-dev-cagra-milvuspoctest2-

[kdabbir]

I got it working, the documentation needs to be updated, basically I had to add below config:

    gpu:
      enable: true
      cache.enable: true
      initMemSize: 0 # Gpu Memory Pool init size
      maxMemSize: 0 # Gpu Memory Pool Max size

gpu - enable:true is needed to enable GPU i believe, please update this documentation to add this configuration: https://milvus.io/docs/index-with-gpu.md#Configure-Milvus-settings-for-GPU-memory-control