Comments (12)
Hi @YaYaB here is a fix that makes your example work here #1210
...
shortest patches sometimes are the longest to produce
from deepdetect.
Hi, have you tried with / without CUDNN ? CUDNN non deterministically chooses internal algorithms depending on batch size. The non deterministic component comes from rolling dice on some decisions, AFAIK.
The fact that there's no issue with classification models may rule this out, but it is still worth testing.
@fantes if not CUDNN, it may be in one of the bbox output layers ? (e.g. DetectionOutput ?).
from deepdetect.
I tried with a cpu image but it can not even load the model because of a caffe error (refinedet can not be used with cpu only?). I am building a version without cudnn to see if that works but I am not sure if I will be able to create the service before trying predictions.
[2021-02-24 15:01:41.628] [caffe] [error] Layer conv1_1 has unknown engine.
[2021-02-24 15:01:41.638] [face_512] [error] Error creating network
[2021-02-24 15:01:41.640] [face_512] [error] service creation call failed: Dynamic exception type: CaffeErrorException
std::exception::what: /opt/deepdetect/build/caffe_dd/src/caffe_dd/include/caffe/llogging.h:194 / Fatal Caffe error
from deepdetect.
maybe you built w/o cpu but with USE_CUDNN=ON , in case of refinedet and CUDNN set to on at compile time, we force cudnn engine, but if it is not built into caffe it will fail. Could you double check that ?
from deepdetect.
The caffe gpu build without CUDNN just finished and I get tthe same error when trying to create the service. I've check that I specified DUSE_CUDNN=OFF
that was also printed when building the image.
[2021-02-24 17:09:07.103] [caffe] [error] Layer conv1_1 has unknown engine.
[2021-02-24 17:09:07.104] [face_512] [error] Error creating network
[2021-02-24 17:09:07.104] [face_512] [error] service creation call failed: Dynamic exception type: CaffeErrorException
std::exception::what: /opt/deepdetect/build/caffe_dd/src/caffe_dd/include/caffe/llogging.h:194 / Fatal Caffe error
from deepdetect.
Hi @YaYaB
did you clean up your repo between your tests w/o cudnn ? maybe your first tests added cudnn engine param in deploy.prototxt (you can search "engine" in deploy.prototxt to check out) and subsequent tests do not support it (because of no cudnn at compile time in dede/caffe ). When cudnn is not compiled in dede, this keyword should not be added (but it is not removed if present, we did not think about this case) . (I'd like to solve this engine problem before looking at the main issue)
from deepdetect.
@fantes Ah indeed, the cudnn engine is indicated in the deploy.prototxt, my bad I should have checked this. I've removed it and it runs correctly. However a more verbose error indicated that the deploy is set for cudnn could be helpful to debug :)
Then, without cudnn I still obtain differences.
Number of documents: v1(98), v2(98)
Number of predictions: 98
1_diff_mean 0.0004954249122921301
1_diff_std 0.004904456410707221
1_diff_max 0.048551641404628754
1_diff_min 0.0
from deepdetect.
ok, thx, I am starting investigations
from deepdetect.
Nice, It happens to everyone then ^^
I'll try it first thing tomorrow, thanks @fantes
from deepdetect.
Congrats to both of you @YaYaB and @fantes for catching this one! Good detailed report + careful inspection are rewarded!
from deepdetect.
from deepdetect.
Sorry for the late answer. It seems to work correctly on my side, cheers @fantes @beniz
from deepdetect.
Related Issues (20)
- Memory leak on constant /predict requests HOT 8
- Refinedet Tensorrt prediction fails HOT 7
- Memory leak on compressed predict requests with oatpp HOT 7
- Different prediction with tensorrt on refinedet model for the version v0.18.0 HOT 3
- getting error while training, .solverstate HOT 23
- Chain predictions swapped between images HOT 2
- Simsearch query segfault when using IVF indexes, but not default/flat index HOT 6
- On object detect training call, missing either test or train list causes a segfault
- dd_client not find in this path anyone help HOT 2
- How do I do a face recognition using this? HOT 2
- DeepDetect full rewrite in Pure Java
- 'OCR' object has no attribute 'histogram_equalization' HOT 13
- "best: -1" in predict behaves differently in torch models HOT 2
- Torch v1.12 requires libcupti* but nvidia/cuda:11.6.0-cudnn8-runtime-ubuntu20.04 doesn't include it
- Race condition / pthread error when predicting
- I have error build xgboost HOT 1
- Using `true` or `false` instead of `1` or `0` for query params for status or labels returns a internal server error HOT 1
- Question about hosting the docker image HOT 4
- Graphics problem with tsne algorithm HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deepdetect.