Comments (15)
I have tried your python code, and the index build can finish successfully. What is your runtime environment?
from sptag.
set n = 1024×1024×3 also can cause this error, but n = 1024×1024×2 is ok.
I dont know why? is there still has wrapper bug ?
from sptag.
I has test to build index with indexbuilder.exe last night, it seems that when the size of raw data large than 4G (eg 6G), the program will fail to complete , stop at "Save Data To xxxx\vectors.bin" . and I open the output directory , there were a vectors.bin which size reached to 300G !
The above phenomenon indicates that the problem is not caused by Python's package interface.
But what is the real reason?
Note: In my test, the memory usage will reach 20G or more, and the establishment of the entire index takes a long time (up to several hours), so if you are not testing a very large amount of data, you will not encounter this problem.
from sptag.
This is a screenshot of the problem:
indexbuilder.exe -d 512 -v float -i BIN:vv.bin -o test1 -a BKT -m Cosine
from sptag.
I am using SPTAG source code compiled with python3.6 64-bit VS2017.
And I run the code under Windows 7 sp1, Python 3.6 64bit。
from sptag.
I have tried your python code, and the index build can finish successfully. What is your runtime environment?
The same problem will occur when running the above code under WINDOWS10 ,python3.6 64bit.
from sptag.
yep, It takes so long to complete the training index
I've tried on the experiment with the dataset (1024 * 1024 * 5, 128) -- 5.242.880 vectors of 128 dim.
Env:
- Core [email protected], 8GRam.
- Python 2.7
-> do: Test('BKT', 'L2')
It took me around 8-9 hours to complete the training. @MaggieQi
from sptag.
yep, It takes so long to complete the training index
I've tried on the experiment with the dataset (1024 * 1024 * 5, 128) -- 5.242.880 vectors of 128 dim.
Env:
- Core [email protected], 8GRam.
- Python 2.7
-> do: Test('BKT', 'L2')
It took me around 8-9 hours to complete the training. @MaggieQi
What‘s your OS version? Linux or Windows?
from sptag.
yep, It takes so long to complete the training index
I've tried on the experiment with the dataset (1024 * 1024 * 5, 128) -- 5.242.880 vectors of 128 dim.
Env:
- Core [email protected], 8GRam.
- Python 2.7
-> do: Test('BKT', 'L2')
It took me around 8-9 hours to complete the training. @MaggieQiWhat‘s your OS version? Linux or Windows?
Linux, Ubuntu 18.04 64bit
from sptag.
@deepxuexi I also tried your python code on Windows server with VS 2015 using the newest version of code. It can also finish successfully. Maybe you can pull the newest code and try it again.
from sptag.
@deepxuexi I also tried your python code on Windows server with VS 2015 using the newest version of code. It can also finish successfully. Maybe you can pull the newest code and try it again.
I compiled SPTAG again with windows7 +VS2015 +tbb4.4+cmake3.14.4+swigwin4.0. The same error occurred during the running process: When there are too many vectors, the saved vectors.bin file will reach 300G.
Can you tell me your detailed compilation environment? Such as the version of the Windows operating system, VS2015 version, cmake, swig, TBB version?
from sptag.
@deepxuexi I also tried your python code on Windows server with VS 2015 using the newest version of code. It can also finish successfully. Maybe you can pull the newest code and try it again.
This is my compiled file, you can run TEST_ERROR.py directly in python3.6 to reproduce the error.
SPTAG_PY36.zip
I wonder if you can send me a copy of your compiled files (only those exe and pyd generated by SPTAG and your tbb.dll), thank you!
from sptag.
@deepxuexi my environment is Windows 10 (also tried it on Windows server 2016) + VS2015 + cmake 3.12.3 + swig 3.0.12 + boost 1.67.0. Which version of SPTAG code do you use? Do you use cmake to compiling the code or directly use the SPTAG.sln?
from sptag.
@deepxuexi my environment is Windows 10 (also tried it on Windows server 2016) + VS2015 + cmake 3.12.3 + swig 3.0.12 + boost 1.67.0. Which version of SPTAG code do you use? Do you use cmake to compiling the code or directly use the SPTAG.sln?
I use the SPTAG code of 2019-5-20,and compling the code with "ALL_BUILD.vcxproj" in VS2015。
from sptag.
@deepxuexi my environment is Windows 10 (also tried it on Windows server 2016) + VS2015 + cmake 3.12.3 + swig 3.0.12 + boost 1.67.0. Which version of SPTAG code do you use? Do you use cmake to compiling the code or directly use the SPTAG.sln?
I just tryed compling the code with VS2015 + cmake 3.12.3 + swig 3.0.12 + boost 1.67.0 and python3.6 ,but the problem remain.
Could you tell me what's your TBB and python version?
thank you very much!
from sptag.
Related Issues (20)
- Logger is not configurable until after it's been used
- SPATAG build failure with cmake HOT 3
- Not enough memory on host devices offered by Azure
- thread local context (#359) causes test issue on Linux: `1: [4] fid:0 channel 2, to submit:64, submitted:Operation not permitted` HOT 18
- This repo is missing important files
- build failure with GCC 13 due to missing `#include <cstdint> in `AnnService/inc/Helper/DiskIO.h`
- index.Save function doesn't create the file on disk HOT 1
- How to Search SPANN SSD Index???
- Import SPTAG Failed
- Missing m_SPTQueue.insert()?
- [QUESTION] How to start a online server for service.ini?
- Multiple connection for one server (to be assigned for each client)
- How to use distribute server? HOT 2
- compile failed with bug
- KMeans clustering
- Improve the BalancedDataPartition program
- Building a 1000W BKT index crashes
- Unable to download vectors_9.bin and vectors_12.bin using git lfs pull
- double free or corruption (out) during Search
- Want to know some features of sptag
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sptag.