Comments (14)
How many index node you are using? and what is the hardware config?
For IVF PQ, latest intel cpu can build one segmnt(1M) in less than i minutes.
where is the link of your baidu net disk?
from milvus.
Milvus_log.txt link:https://pan.baidu.com/s/1nXvr3DMLe8_2A_Q-F7E6iw?pwd=x042
password:x042
from milvus.
/assign @xiaocai2333
/unassign
from milvus.
are u using intel cpu?
from milvus.
- did you try IVFSQ8? SQ should be much faster than PQ.
- can you add more index node and try
from milvus.
@JeremyLee-lrj can you send the log to my email: [email protected]
?
from milvus.
are u using intel cpu?
from milvus.
- did you try IVFSQ8? SQ should be much faster than PQ.
- can you add more index node and try
- No, I haven't try IVFSQ8, thanks for sharing it, I think I will try IVF_SQ later.
- I'm sorry that I only have one server right now, So I think I can't add more index node.
from milvus.
@JeremyLee-lrj can you send the log to my email:
[email protected]
?
Sure, I'm sending it right now, but it will take some time to upload the milvus_log.txt to mail.
from milvus.
@JeremyLee-lrj Sorry for late reply.
IVF_PQ is inherently a slow index to build. From the logs you provided, I didn't find any abnormal situations; everything is proceeding normally. I also tested locally, and it indeed takes over 10 minutes to build an index for 200000 data points with 256 dimensions. If you want to build an index quickly, I recommend using HNSW, which also has excellent recall performance.
from milvus.
IVF_SQ8 could be much faster as well.
from milvus.
@JeremyLee-lrj Sorry for late reply. IVF_PQ is inherently a slow index to build. From the logs you provided, I didn't find any abnormal situations; everything is proceeding normally. I also tested locally, and it indeed takes over 10 minutes to build an index for 200000 data points with 256 dimensions. If you want to build an index quickly, I recommend using HNSW, which also has excellent recall performance.
It's OK, I'm fine with the reply time, and I appreciate your help very much!
Actually, I use IVF_PQ because I want smaller index size, but I didn't realize the building time is so long. Thanks for your recommendation!
from milvus.
IVF_SQ8 could be much faster as well.
OK, I got it, appreciate it!
from milvus.
i'd close this issue, and please feel free to file an new one if any other issues
from milvus.
Related Issues (20)
- [Bug]: [GO SDK] search raises error when filter by scalar fields which builds BITMAP index with retrieve data HOT 3
- [Enhancement]: L0 compaction related UT cov to 90% HOT 1
- [Bug]: [GO SDK] search raises error ` query failed: Operator::GetOutput failed` when filter by `array` filed which build BITMAP index and `int` field which not build index HOT 4
- [Enhancement]: Refine querycoord task schedule
- [Feature]: Case insensitive varchar querying HOT 3
- [Enhancement]: Unify the generation and conversion of pchannel and vchannel
- [Bug]: When a request fails, some processes are wrong, such as logs and metrics. HOT 1
- [Bug]: failed to gpubuilder make with grpc error HOT 2
- [Bug]: search_iterator taking too much time to give response after some iteration HOT 11
- [Bug]: incomplete query result, missing id HOT 8
- [Bug]: Milvus Standalone container randomly stopping + Connection Failed HOT 11
- [Bug]: [Bug]:failed to search/query delegator 617 for channel by-dev-rootcoord-dml_4_448959102009636230v0: fail to Search, QueryNode ID=617, reason=Timestamp lag too large HOT 2
- [Bug]: unstable ut of compaction pickSlot HOT 3
- [Bug]: unstable ut test case parseIndexParams HOT 1
- [Bug]: unstable compaction UT TestPickAnyNode HOT 1
- [Bug]: NULL Pointer Dereference Vulnerability HOT 2
- [Bug]: [Nightly] Search failed reporting GetOutput failed when existing index on varchar field HOT 1
- [Bug]: RSS memory too high in datanode when continuously create collections and insert data HOT 7
- [Bug]: collection meta isn't removed in dataCoord after collection GC completed HOT 5
- [Bug]: milvus standalone crash by bitset assert check fail HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from milvus.