Comments (6)
First investigation reveals this is faiss crashing:
Thread 37 "dede" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffe50ffd700 (LWP 10616)]
0x0000555555c49c41 in faiss::OnDiskInvertedLists::OngoingPrefetch::Thread::one_list() ()
(gdb) bt
#0 0x0000555555c49c41 in faiss::OnDiskInvertedLists::OngoingPrefetch::Thread::one_list() ()
#1 0x0000555555c49d7b in faiss::OnDiskInvertedLists::OngoingPrefetch::prefetch_list(void*) ()
#2 0x00007fffda3366db in start_thread (arg=0x7ffe50ffd700) at pthread_create.c:463
#3 0x00007fffd20aa71f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
from deepdetect.
I tried indexing 600 images, with a much lower train_samples
and nprobe
but the same problem..
"index_type": "IVF20,SQ8",
"train_samples": 10,
"nprobe": 8
in model/
-rw-r--r-- 1 dgtlmoon dgtlmoon 181948 Nov 15 08:49 index.faiss
-rw-r--r-- 1 dgtlmoon dgtlmoon 4194304 Nov 15 08:49 index_mmap.faiss
[2021-11-15 07:51:19.101] [torchlib] [info] Network initialization done.
[2021-11-15 07:51:19.102] [test] [info] Using pre-trained weights from /var/www/xxx/web/files/trainer/simsearch/model/model_iter_13500.caffemodel
[2021-11-15 07:51:19.408] [torchlib] [info] Attempting to upgrade batch norm layers using deprecated params: /var/www/xxx/web/files/trainer/simsearch/model/model_iter_13500.caffemodel
[2021-11-15 07:51:19.408] [torchlib] [info] Successfully upgraded batch norm layers using deprecated params.
[2021-11-15 07:51:19.512] [test] [info] Net total flops=3858534272 / total params=26063936
[2021-11-15 07:51:19.512] [test] [info] detected network type is classification
[2021-11-15 07:51:19.513] [api] [info] HTTP/1.1 "PUT /services/test" <n/a> 201 508ms
open existing index db
[2021-11-15 07:51:23.846] [torchlib] [info] Opened lmdb /var/www/xxx/web/files/trainer/simsearch/model//names.bin
bash: line 1: 7 Segmentation fault (core dumped) ./dede -host 0.0.0.0
from deepdetect.
Maybe the version of FAISS changed, and it worked with an earlier lib version?
from deepdetect.
The cmake doesnt seem to be tied to any particular version
message(STATUS "Fetching FAISS")
add_definitions(-DUSE_FAISS)
add_definitions(-DUSE_SIMSEARCH)
ExternalProject_Add(
faisslib
PREFIX faiss
SOURCE_DIR ${CMAKE_BINARY_DIR}/faiss/src/faiss
GIT_REPOSITORY https://github.com/facebookresearch/faiss
GIT_TAG main
UPDATE_DISCONNECTED 1
CONFIGURE_COMMAND cmake ${CONFIGURE_OPTS} -B build .
BUILD_COMMAND make -C build -j${N}
INSTALL_COMMAND ""
BUILD_IN_SOURCE 1
)
from deepdetect.
This is fixed with #1369, disabling MMAP, no alternative solution found at the moment :/ Thanks for the report.
from deepdetect.
@beniz no probs, this will make its way into a docker build at some time?
from deepdetect.
Related Issues (20)
- Inconsistent predictons using refinedet model HOT 12
- Memory leak on constant /predict requests HOT 8
- Refinedet Tensorrt prediction fails HOT 7
- Memory leak on compressed predict requests with oatpp HOT 7
- Different prediction with tensorrt on refinedet model for the version v0.18.0 HOT 3
- getting error while training, .solverstate HOT 23
- Chain predictions swapped between images HOT 2
- On object detect training call, missing either test or train list causes a segfault
- dd_client not find in this path anyone help HOT 2
- How do I do a face recognition using this? HOT 2
- DeepDetect full rewrite in Pure Java
- 'OCR' object has no attribute 'histogram_equalization' HOT 13
- "best: -1" in predict behaves differently in torch models HOT 2
- Torch v1.12 requires libcupti* but nvidia/cuda:11.6.0-cudnn8-runtime-ubuntu20.04 doesn't include it
- Race condition / pthread error when predicting
- I have error build xgboost HOT 1
- Using `true` or `false` instead of `1` or `0` for query params for status or labels returns a internal server error HOT 1
- Question about hosting the docker image HOT 4
- Graphics problem with tsne algorithm HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deepdetect.