Comments (19)
I am still getting this issue on version 2.0.0
from realtime_object_detection.
@gustavz now i install tensorflow1.7 on jetson tx2 but face same issue:
Loading label map
Building Graph
2018-03-29 01:16:27.067350: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
2018-03-29 01:16:27.067445: E tensorflow/core/common_runtime/direct_session.cc:167] Internal: CUDA runtime implicit initialization on GPU:0 failed. Status: unknown error
Traceback (most recent call last):
File "object_detection.py", line 301, in
main()
File "object_detection.py", line 297, in main
detection(graph, category, score, expand)
File "object_detection.py", line 180, in detection
with tf.Session(graph=detection_graph,config=config) as sess:
File "/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1509, in init
super(Session, self).init(target, graph, config=config)
File "/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 638, in init
self._session = tf_session.TF_NewDeprecatedSession(opts, status)
File "/home/nvidia/.local/lib/python2.7/site-packages/tensorflow/python/framework/errors_impl.py", line 516, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InternalError: Failed to create session.
can you suggest how to do?
from realtime_object_detection.
Hi @gustavz,
I searched about multiple session problem. here:
https://devtalk.nvidia.com/default/topic/1035884/jetson-tx2/cuda-error-creating-more-than-one-session-using-tensorflow/post/5265161/#5265161
We need to add gpu_options in the tf.Session() that is called at the first.
v1.0: object_detection.py
v2.0: rod/model.py
def load_frozenmodel():
...
input_graph = tf.Graph()
config = tf.ConfigProto()
config.gpu_options.allow_growth = allow_memory_growth
with tf.Session(graph=input_graph, config=config):
from realtime_object_detection.
I'm using the latest version of tensorflow (2.3.0) with python 3.6.10 and cuda 10.1 and facing the same issue as well in Ubuntu 18.04. export CUDA_VISIBLE_DEVICES=0 or export CUDA_VISIBLE_DEVICES='' are helping to run the code but not consistent enough. I'm new and don't actually know what these exports are doing exactly. This is the error I get:
RuntimeError: CUDA runtime implicit initialization on GPU:0 failed. Status: device kernel image is invalid
from realtime_object_detection.
Limiting the GPU memory solves this issue for me:
import tensorflow as tf
MEMORY_LIMIT = 1024
gpus = tf.config.experimental.list_physical_devices('GPU')
if gpus:
try:
tf.config.experimental.set_virtual_device_configuration(gpus[0], [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=MEMORY_LIMIT)])
except RuntimeError as e:
print(e)
from realtime_object_detection.
Okey, downgrading tensorflow to version 2.2 did it.
pip install --force-reinstall tensorflow-gpu==2.2
Also, if you have ever used pip install
with --ignore-installed
to install tensorflow versions or dependencies, consider removing them first.
What's strange for me is that it takes some minutes to initialize tensorflow with GPU. I don't think it's normal :S
from realtime_object_detection.
I had a similar issue. I was using tensorflow 1.5. I downgraded to 1.4.1 and now it works.
from realtime_object_detection.
yeah thats tensorflow version bound.
The reason is the version the model is exported in.
There are incompabilities between 1.5 and 1.4
from realtime_object_detection.
@MatthiasRoelandts does the error still appear?
from realtime_object_detection.
@imxboards i also sometimes face this error when i switch tensorflow versions.
It is definetly caused by tensorflows internal changes and in-compabilities.
Try using TF-1.4.
I for my self am also not able to run it with TF-1.7
from realtime_object_detection.
I'm experiencing this exact same issue. I had my code up and running on the Jetson TX2 with tensorflow 1.7, but now after powering down and traveling for a few days, it's giving me this error. I can run it without gpu using export CUDA_VISIBLE_DEVICES=''
however this defeats the purpose. Anybody have any solutions? I could try going to an older version of TF, however all the Jetson builds/wheel files I can find are for Python 2.7 whereas my entire project is written in Python 3.5. Any help would be greatly appreciated!
from realtime_object_detection.
@ibeckermayer - it is often surprisingly easy to get a Python program running under both Python 3.5 and 2.7. Only took me 30 minutes to convert my 3.5 program - I only had issues where I was using datetime
routines to generate utc time.
from realtime_object_detection.
@naisy which problem should this solve?
from realtime_object_detection.
This solves the problem in environments where 2nd session can not be created.
tensorflow.python.framework.errors_impl.InternalError: Failed to create session.
from realtime_object_detection.
I am using tensotflow 2.0.0 in c++ code and am facing the same issue. Any solution guys?
from realtime_object_detection.
@ctsams9 I am having a same situation as yours. Did you find any solution so far ?
from realtime_object_detection.
@ctsams9 I am having a same situation as yours. Did you find any solution so far ?
@horcrux1 No luck so far. I'm planning to post some of the results I get after running some tests here and on stackoverflow soon.
from realtime_object_detection.
I'm using the latest version of tensorflow (2.3.0) with python 3.6.10 and cuda 10.1 and facing the same issue as well in Ubuntu 18.04. export CUDA_VISIBLE_DEVICES=0 or export CUDA_VISIBLE_DEVICES='' are helping to run the code but not consistent enough. I'm new and don't actually know what these exports are doing exactly. This is the error I get:
RuntimeError: CUDA runtime implicit initialization on GPU:0 failed. Status: device kernel image is invalid
Just clarifying @ctsams9 exports, both of them set the variable CUDA_VISIBLE_DEVICES, which is useful if you only want cuda/tensorflow to see and work with specific GPU (0 equals the first GPU). If this variable is set to "", then tensorflow will only use CPU for calculations.
In my case, it runs fine without letting my only GPU (0) visible, but I need to fix whatever "device kernel image is invalid" error means to proceed with my projects.
By the way, I am getting the same error on Arch after upgrading tensorflow from pip.
from realtime_object_detection.
Limiting the GPU memory solves this issue for me:
import tensorflow as tf MEMORY_LIMIT = 1024 gpus = tf.config.experimental.list_physical_devices('GPU') if gpus: try: tf.config.experimental.set_virtual_device_configuration(gpus[0], [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=MEMORY_LIMIT)]) except RuntimeError as e: print(e)
I tried to limit TF GPU memory earlier but had no success, sadly :(
from realtime_object_detection.
Related Issues (20)
- is it possible to input a gray-style video? HOT 1
- Half FPS using tensorflow's official .pb file HOT 1
- low fps on tx2 is only 0.5 HOT 20
- Running run_objectdetection.py results in indentation error HOT 3
- Does not work with Tensorflow 1.8 HOT 11
- run_objectdetection can not be run successfully HOT 5
- High inference time using r1.0 and master HOT 3
- Any plan to upgrade object detection model to mobilenetV2-ssdlite? HOT 7
- The problem of running objectdetection_image.py when MULTI_THREADING: True
- Trying to use the GPU on my computer HOT 1
- No bounding boxes with deeplab_video.py HOT 2
- License for repository HOT 2
- Can I change the environment?I must need python3.6 and tf1.8 HOT 1
- Hi Gustav I have questions! HOT 1
- How can I fix it? HOT 5
- Query on Performance of r1.0 version of the code HOT 1
- Does this repo support Faster-RCNN network with Jetson-TX2 in tensor-RT related stuffs?
- How to convert the Mask RCNN model to tflite model HOT 1
- RuntimeError: Attempted to use a closed Session.
- Node 'Pad_17': Unknown input node 'Relu_18' when call import_graph_def for pb file HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from realtime_object_detection.