Comments (21)
please send it to my mail: [email protected]
from milvus.
I guess the empty segments actually are not "empty", they are compacted segments and pending for indexing. In Milvus 2.3.12, the default maxSize of a segment is 512MB, while in Milvus 2.4.4 it changed to 1GB, so after upgraded, Milvus is trying to compact the segments to a the new maxSize. If you checking the compaction tasks or indexing tasks, I believe you can see the new tasks.
BTW, a bigger segment size leads to a better search performance in general. But it is not the bigger the better, because a bigger segment also means lower efficiency usage in memory.
/assign @Archalbc
/unassign
from milvus.
I guess the empty segments actually are not "empty", they are compacted segments and pending for indexing. In Milvus 2.3.12, the default maxSize of a segment is 512MB, while in Milvus 2.4.4 it changed to 1GB, so after upgraded, Milvus is trying to compact the segments to a the new maxSize. If you checking the compaction tasks or indexing tasks, I believe you can see the new tasks. BTW, a bigger segment size leads to a better search performance in general. But it is not the bigger the better, because a bigger segment also means lower efficiency usage in memory.
/assign @Archalbc /unassign
Hey sorry I completely forgot to say that we are already at 2GB for the maxSegmentSize since 2.3.12 as our querynodes are 32GB. (I'm updating the initial post)
from milvus.
trying to reproduce the issue in house.... @Archalbc any configs else did you change for milvus cluster
/assign @Archalbc
from milvus.
trying to reproduce the issue in house.... @Archalbc any configs else did you change for milvus cluster /assign @Archalbc
Hello, here is my values.yaml for the milvus configuration:
extraConfigFiles:
user.yaml: |+
queryNode:
scheduler:
maxReadConcurrentRatio: 2
dataCoord:
segment:
maxSize: 2048
queryCoord:
overloadedMemoryThresholdPercentage: 80
vector dimension: 256
HNSW : ef: 100, efContruction: 360, M: 8
The client calls both query() and search() interfaces, fields : embedding and createdAt.
4 shards per collection
from milvus.
@Archalbc
are you using bulkinsertion?
#33604 this might be the issue related.
We will fix this soon
from milvus.
@Archalbc are you using bulkinsertion? #33604 this might be the issue related. We will fix this soon
My workmate will anwser to you soon.
from milvus.
New event, after the downgrade to 2.3.17 (so finally I upgraded from 2.3.12 to 2.3.17). Everything was normal until last friday around 1h AM, we encountered again a huge quantities of segments created. But this time, I can't see those segments on ATTU or birdwatcher :o, they are telling me I should have around 100 sealed segments, but this is clearly not the case.
In 2.4.4 I was able to see those empty segments on ATTU.
Birdwatcher, right now is saying;
--- Growing: 32, Sealed: 0, Flushed: 68, Dropped: 19
--- Total Segments: 100, row count: 65129835
and ATTU for the collection ID 450234905777292996:
while the Sealed segment vizualization is saying more than 70 segments. As you can see, before the sudden increase of sealed segments, I should have 40 segments for this collection, which is normal because we have 2 replicas.
I rollback again to 2.3.12, And I got my 40 sealed segments again...
from milvus.
New event, after the downgrade to 2.3.17 (so finally I upgraded from 2.3.12 to 2.3.17). Everything was normal until last friday around 1h AM, we encountered again a huge quantities of segments created. But this time, I can't see those segments on ATTU or birdwatcher :o, they are telling me I should have around 100 sealed segments, but this is clearly not the case. In 2.4.4 I was able to see those empty segments on ATTU.
Birdwatcher, right now is saying;
--- Growing: 32, Sealed: 0, Flushed: 68, Dropped: 19 --- Total Segments: 100, row count: 65129835
and ATTU for the collection ID 450234905777292996:
while the Sealed segment vizualization is saying more than 70 segments. As you can see, before the sudden increase of sealed segments, I should have 40 segments for this collection, which is normal because we have 2 replicas.
I rollback again to 2.3.12, And I got my 40 sealed segments again...
- did you have any import and delete at that time?
- you have 32 growing segments, which means you might have 32 partitions or enabled partitionkey features. If you have many partitions, has hundred segments might not be a big thing?
from milvus.
because each time you flush(or auto flush), your segment number will grow by 32. and after compaction the number will decrease
from milvus.
- We did nothing special during the night, just usual operations.
- We always have only one partition per collection.
We don't call flush in the code, we let the system auto-flush.
There is definitively something strange because reverting back to 2.3.12, odd segments disappear and we are back to the normal number of segments.
We can clearly see that those strange segments are never flushed or compacted, they stay here forever...
from milvus.
- We did nothing special during the night, just usual operations.
- We always have only one partition per collection.
We don't call flush in the code, we let the system auto-flush. There is definitively something strange because reverting back to 2.3.12, odd segments disappear and we are back to the normal number of segments. We can clearly see that those strange segments are never flushed or compacted, they stay here forever...
If you only have one collection and one partition, there seems to be no reason you will see 32 growing segments.
did you do import or insert? can you offer full logs for debug?
from milvus.
I think we need logs and also information about "what operation did you do and see a segemnt with 0 entities"
from milvus.
- We did nothing special during the night, just usual operations.
- We always have only one partition per collection.
We don't call flush in the code, we let the system auto-flush. There is definitively something strange because reverting back to 2.3.12, odd segments disappear and we are back to the normal number of segments. We can clearly see that those strange segments are never flushed or compacted, they stay here forever...
If you only have one collection and one partition, there seems to be no reason you will see 32 growing segments.
did you do import or insert? can you offer full logs for debug?
No we have 8 collections, since the rollback to 2.3.12, we have the correct number of segments and it's not growing.
I'm starting to wonder if this is because client are only using SDK Go 2.3.2 ?
Sorry I cannot provide any logs since each time I rollback ASAP and our logging system is currently under maintenance u_u.
My workmate is not available today to give more insight on the client side.
Maybe we will see with the team if we can somehow reproduce that in another cluster.
from milvus.
I guess this might not be related to SDK version. But we need more clues about why the log is created
from milvus.
@Archalbc I did not reproduce this issue in my milvus, but i can try more time. If it reproduced to you, could you please attach the etcd backup for investigation? Check this: https://github.com/milvus-io/birdwatcher for details about how to backup etcd with birdwatcher.
One more question is could you please double check with your workmates that there is no insert or delete operations before/during the system upgrading?
from milvus.
Hi,
Sorry for late answer
are you using bulkinsertion?
#33604 this might be the issue related.
We will fix this soon
Sorry for late answer !! We're not using bulk insert. We're upserting each time
from milvus.
upsert seems to be the issue. becasue upsert also cause one delete on each search.
The segment is under L0 state. If you rollback your cluster version to 2.3, you might lost some deletes in L0 segments(because 2.3 don't recommend these deletes)
The segment number increase might be also related to L0 delete, but this might be as expected and won't affect search performance
from milvus.
upsert seems to be the issue. becasue upsert also cause one delete on each search. The segment is under L0 state. If you rollback your cluster version to 2.3, you might lost some deletes in L0 segments(because 2.3 don't recommend these deletes)
The segment number increase might be also related to L0 delete, but this might be as expected and won't affect search performance
I can understand that upsert operations are kinda tricky, but I don't understand why we have a change of behavior on the cluster. Everything look fine in 2.3.12 and I doubt the cluster will be OK with an infinite number of segments (because you can see in the graph that it never stop creating segments and they are never cleaned.)
How can I securely provide you the etcd backup without exposing it here ?
from milvus.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen
.
from milvus.
/assign @congqixia
please keep an eye on it
from milvus.
Related Issues (20)
- [Enhancement]: Unify the generation and conversion of pchannel and vchannel
- [Bug]: When a request fails, some processes are wrong, such as logs and metrics. HOT 1
- [Bug]: failed to gpubuilder make with grpc error HOT 2
- [Bug]: search_iterator taking too much time to give response after some iteration HOT 11
- [Bug]: incomplete query result, missing id HOT 8
- [Bug]: Milvus Standalone container randomly stopping + Connection Failed HOT 11
- [Bug]: [Bug]:failed to search/query delegator 617 for channel by-dev-rootcoord-dml_4_448959102009636230v0: fail to Search, QueryNode ID=617, reason=Timestamp lag too large HOT 2
- [Bug]: unstable ut of compaction pickSlot HOT 3
- [Bug]: unstable ut test case parseIndexParams HOT 1
- [Bug]: unstable compaction UT TestPickAnyNode HOT 1
- [Bug]: NULL Pointer Dereference Vulnerability HOT 1
- [Bug]: [Nightly] Search failed reporting GetOutput failed when existing index on varchar field HOT 1
- [Bug]: RSS memory too high in datanode when continuously create collections and insert data HOT 5
- [Bug]: collection meta isn't removed in dataCoord after collection GC completed HOT 5
- [Bug]: milvus standalone crash by bitset assert check fail HOT 2
- [Bug]: Can't use MilvusClient() because it says "ModuleNotFoundError: No module named 'milvus_lite'" HOT 5
- [Enhancement]: Under standalone mode, limit all node cpu and memory usage.
- [Bug]: clustering compaction collectionIsCompacting logic is wrong HOT 2
- [Enhancement]: Show stack backtrace info if milvus crashes with signal generated from non-go thread
- [Enhancement]: use the collection id to group msg pack in the msg dispatcher
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from milvus.