[Bug]: standalone panic with error `panic: segment not found[segment=450570869009695178]` during test after reinstallation or upgrade

[zhuwenxing]

[Bug]: standalone panic with error `panic: segment not found[segment=450570869009695178]` during test after reinstallation or upgrade

[zhuwenxing]

It also reproduced after v2.3.15 --> master-20240619-7b9462c0-amd64

error

[2024/06/19 10:09:59.079 +00:00] [INFO] [metacache/meta_cache.go:299] ["remove dropped segment"] [segmentID=450570799631185937]
[2024/06/19 10:09:59.079 +00:00] [INFO] [datacoord/session_manager.go:256] ["success to sync segments"] [nodeID=3] [planID=0]
[2024/06/19 10:09:59.079 +00:00] [INFO] [datacoord/sync_segments_scheduler.go:145] ["sync segments success"] [collectionID=450570409833736812] [partitionID=450570409833736813] [channelName=by-dev-rootcoord-dml_2_450570409833736812v1] [nodeID=3] [segments="[450570799631185937,450570799631185738,450570799630779282]"]
[2024/06/19 10:09:59.080 +00:00] [INFO] [datanode/services.go:284] ["DataNode receives SyncSegments"] [traceID=a004a4287697927e184d13448af58e44] [planID=0] [nodeID=3] [collectionID=450570409833736906] [partitionID=450570409833736907] [channel=by-dev-rootcoord-dml_8_450570409833736906v0]
[2024/06/19 10:09:59.080 +00:00] [INFO] [util/load_stats.go:38] ["begin to init pk bloom filter"] [segmentID=450570799630779283] [statsBinLogsLen=1]
[2024/06/19 10:09:59.085 +00:00] [INFO] [util/load_stats.go:113] ["Successfully load pk stats"] [segmentID=450570799630779283] [time=5.178694ms] [size=408064]
panic: segment not found[segment=450570799632187110]

goroutine 18885 [running]:
panic({0x5690c60?, 0xc0084c5680?})
	/usr/local/go/src/runtime/panic.go:1017 +0x3ac fp=0xc005261540 sp=0xc005261490 pc=0x1e27b2c
[2024/06/19 10:09:59.085 +00:00] [INFO] [metacache/meta_cache.go:289] ["metacache does not have segment, add it"] [segmentID=450570799630779283]
github.com/milvus-io/milvus/internal/datanode/syncmgr.(*storageV1Serializer).setTaskMeta.func1({0x5f69ea0?, 0xc0084c5680?})
	/go/src/github.com/milvus-io/milvus/internal/datanode/syncmgr/storage_serializer.go:158 +0x26 fp=0xc005261560 sp=0xc005261540 pc=0x4620146
github.com/milvus-io/milvus/internal/datanode/syncmgr.(*SyncTask).HandleError(0xc0058c8c60, {0x5f69ea0, 0xc0084c5680})
	/go/src/github.com/milvus-io/milvus/internal/datanode/syncmgr/task.go:117 +0x7c fp=0xc0052615f0 sp=0xc005261560 pc=0x461b99c
[2024/06/19 10:09:59.085 +00:00] [INFO] [metacache/meta_cache.go:299] ["remove dropped segment"] [segmentID=450570409833336946]
[2024/06/19 10:09:59.085 +00:00] [INFO] [metacache/meta_cache.go:299] ["remove dropped segment"] [segmentID=450570409833337817]
[2024/06/19 10:09:59.085 +00:00] [INFO] [metacache/meta_cache.go:299] ["remove dropped segment"] [segmentID=450570409834537791]
[2024/06/19 10:09:59.085 +00:00] [INFO] [datacoord/session_manager.go:256] ["success to sync segments"] [nodeID=3] [planID=0]
[2024/06/19 10:09:59.085 +00:00] [INFO] [datacoord/sync_segments_scheduler.go:145] ["sync segments success"] [collectionID=450570409833736906] [partitionID=450570409833736907] [channelName=by-dev-rootcoord-dml_8_450570409833736906v0] [nodeID=3] [segments="[450570799631185466,450570799631185548,450570799630779283]"]

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/deploy_test_cron/detail/deploy_test_cron/2433/pipeline
log:
artifacts-rocksmq-standalone-upgrade-2433-server-logs.tar.gz

[xiaofan-luan]

/assign @congqixia

please help on it

[congqixia]

The root cause was the SyncSegments call removed a new growing segments from meta, which caused panicking when datanode needs to flush it

[congqixia]

patch merged, could you please helping verify?
/assign @zhuwenxing

[zhuwenxing]

Not reproduced