[Bug]: flush failed with error `can not find session: node not found` after etcd pod failure chaos test

[zhuwenxing]

[Bug]: flush failed with error `can not find session: node not found` after etcd pod failure chaos test

[zhuwenxing]

/assign @XuanYang-cn

[zhuwenxing]

there is also another grpc error

[2024-05-19T19:59:10.906Z] [2024-05-19 19:53:09 - DEBUG - ci_test]: (api_response) : <Collection>:

[2024-05-19T19:59:10.906Z] -------------

[2024-05-19T19:59:10.906Z] <name>: QueryChecker__JyGnax9q

[2024-05-19T19:59:10.906Z] <description>: 

[2024-05-19T19:59:10.906Z] <schema>: {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float', 'description': '', 'type': <DataType.FLOAT:......  (api_request.py:37)

[2024-05-19T19:59:10.906Z] [2024-05-19 19:53:09 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)

[2024-05-19T19:59:10.906Z] [2024-05-19 19:56:09 - ERROR - pymilvus.decorators]: grpc RpcError: [flush], <_MultiThreadedRendezvous: StatusCode.DEADLINE_EXCEEDED, Deadline Exceeded>, <Time:{'RPC start': '2024-05-19 19:53:09.557337', 'gRPC error': '2024-05-19 19:56:09.558691'}> (decorators.py:150)

[2024-05-19T19:59:10.906Z] [2024-05-19 19:56:09 - ERROR - ci_test]: Traceback (most recent call last):

[2024-05-19T19:59:10.906Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper

[2024-05-19T19:59:10.906Z]     res = func(*args, **_kwargs)

[2024-05-19T19:59:10.906Z]   File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request

[2024-05-19T19:59:10.906Z]     return func(*arg, **kwargs)

[2024-05-19T19:59:10.906Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 319, in flush

[2024-05-19T19:59:10.906Z]     conn.flush([self.name], timeout=timeout, **kwargs)

[2024-05-19T19:59:10.906Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 154, in handler

[2024-05-19T19:59:10.906Z]     raise e from e

[2024-05-19T19:59:10.906Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 136, in handler

[2024-05-19T19:59:10.906Z]     return func(*args, **kwargs)

[2024-05-19T19:59:10.906Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 175, in handler

[2024-05-19T19:59:10.906Z]     return func(self, *args, **kwargs)

[2024-05-19T19:59:10.906Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 90, in handler

[2024-05-19T19:59:10.906Z]     raise e from e

[2024-05-19T19:59:10.906Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 86, in handler

[2024-05-19T19:59:10.906Z]     return func(*args, **kwargs)

[2024-05-19T19:59:10.906Z]   File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 1418, in flush

[2024-05-19T19:59:10.906Z]     response = future.result()

[2024-05-19T19:59:10.906Z]   File "/usr/local/lib/python3.8/dist-packages/grpc/_channel.py", line 881, in result

[2024-05-19T19:59:10.906Z]     raise self

[2024-05-19T19:59:10.906Z] grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:

[2024-05-19T19:59:10.906Z] 	status = StatusCode.DEADLINE_EXCEEDED

[2024-05-19T19:59:10.906Z] 	details = "Deadline Exceeded"

[2024-05-19T19:59:10.906Z] 	debug_error_string = "UNKNOWN:Deadline Exceeded {created_time:"2024-05-19T19:56:09.558020013+00:00", grpc_status:4}"

[2024-05-19T19:59:10.906Z] >

[2024-05-19T19:59:10.906Z]  (api_request.py:45)

[2024-05-19T19:59:10.907Z] [2024-05-19 19:56:09 - ERROR - ci_test]: (api_response) : <_MultiThreadedRendezvous of RPC that terminated with:

[2024-05-19T19:59:10.907Z] 	status = StatusCode.DEADLINE_EXCEEDED

[2024-05-19T19:59:10.907Z] 	details = "Deadline Exceeded"

[2024-05-19T19:59:10.907Z] 	debug_error_string = "UNKNOWN:Deadline Exceeded {created_time:"2024-05-19T19:56:09.558020013+00:00", grpc_status:4}"

[2024-05-19T19:59:10.907Z] > (api_request.py:46)

[yanliang567]

/unassign

[XuanYang-cn]

/assign @zhuwenxing
/unassign
Please help verify

[zhuwenxing]

Not reproduced