theaiguyscode/yolov3_deepsort

Object tracking implemented with YOLOv3, Deep Sort and Tensorflow.

yolov3_deepsort's Issues

r

Items are not detecting in yolov3_tiny

I have used the colab version of this code and generate weights with that.

It was detecting correctly with images in google colab, But when I used the same weights with yolov3_deepsort code, This is not detecting anything in the video

I am running this command.

python object_tracker.py --video 0 --output ./data/video/results.avi --weights ./weights/yolov3-tiny.tf --tiny True

How to Detect just 1 class?

I wanted to implement object tracker specifically for one class i.e 'Person'. Is there a way to restrict the number of classes to be detected? so that no extra computational power is used on detecting redundant objects in the input video. What changes should be made in object_detector.py file or detection.py file so that this is possible?

Issue while executing object_tracker.py

I am using Google Colab for my computations. I followed everything as you have mentioned, but encountered this error while executing object_tracker.py

changing the bounding box color

hello
how to change the bounding box color while tracking for example i want it just green..

What device are you using?

I saw your last object tracking video showing that fps almost reach 10 .So,could you tell me what device do you use?Thanks！

Error during execution

after executing:
python object_tracker.py --video 0 --output ./data/video/results.avi

I get following error:

Traceback (most recent call last):
File "object_tracker.py", line 163, in
app.run(main)
File "/home/machine/miniconda3/envs/tracker-gpu/lib/python3.7/site-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/home/machine/miniconda3/envs/tracker-gpu/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "object_tracker.py", line 155, in main
if FLAGS.ouput:
File "/home/machine/miniconda3/envs/tracker-gpu/lib/python3.7/site-packages/absl/flags/_flagvalues.py", line 473, in getattr
raise AttributeError(name)
AttributeError: ouput

Need a list of the person IDs example [8, 9 , 13...] for a project

do you know what to print to generate this said list? for the same video of the pedestrians walking in the streets

non-maxima suppresion after feature generation is computationally wasteful

In object_tracker.py you first do detections = [Detection(bbox, score, class_name, feature) for bbox, score, class_name, feature in zip(converted_boxes, scores[0], names, features)] and then indices = preprocessing.non_max_suppression(boxs, classes, nms_max_overlap, scores)

I suggest that you perform NMS immediately after running yolo, so you don't waste time computing boxes that will just end up unused

How to crop the detetctions?

How to save the detections of each frame? For an example all the frames generated by one tracker object?

log file format

After running the tracker, the bounding boxes are written in a log.txt file. I believe the format is <frame number> <x> <y> <width> <hight>. From which point are the x and y coordinates? top left or center of the bounding box or something else?

Precision Recall curve

Hi!
Is it somehow possible to plot precision recall curve?

No outputs while executing tracker

When I executing the tracker with following code
python3 object_tracker.py --video ./data/video/test.mp4 --output ./data/video/results.avi
There's no output video exists.
How should I fix the problem??

Am having trouble making Gui when import preprocess_image as pi . The output error is ModuleNotFoundError: No module named 'preprocess_image'

Exception in Tkinter callback
Traceback (most recent call last):
File "C:\Anaconda\envs\tensorflowbackend\lib\tkinter_init_.py", line 1705, in call
return self.func(*args)
File "", line 2, in submit
import preprocessing_image as pi
ModuleNotFoundError: No module named ['preprocessing_image']

Could you share your video dataset links?

Wasn't able to run with GPU, but works with CPU

So I faced the issue where only the first frame is detected and tracked when I running the object_tracker.py using tensorflow-gpu. However, once I uninstalled tensorflow-gpu and replaced it with tensorflow-cpu (latest version since I am using python 3.9), it works flawlessly. Does anyone one faced this issue? I am using GTX 1050 on my laptop which I believe it should run since emasterclassacademy (an author who ran this code on YOUTUBE with GTX1050ti) was able to run the code . FYI, I am running on latest CUDA version 11 as well. Below is the full GPU information I have gotten when I ran with tensorflow-gpu.

Using Intel GPU insted of Nvdia

while running the script this is the stat i get

cpu - 98%
ram- almost full
intel gpu-30-40%
nvdia gpu - 3%

my initial conclusion is my code is running on intel gpu
thats why getting 5-6 FPS max

Output video cannot play

Just running the code, the output video is 10 790 bytes and invalid. I guess, no inference is done.

when the code for web cam

Traceback (most recent call last):
File "/content/Single-Multiple-Custom-Object-Detection-and-Tracking/object_tracker.py", line 163, in
app.run(main)
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 303, in run
_run_main(main, args)
File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 251, in _run_main
sys.exit(main(argv))
File "/content/Single-Multiple-Custom-Object-Detection-and-Tracking/object_tracker.py", line 54, in main
yolo.load_weights(FLAGS.weights)
File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/keras/engine/training.py", line 2319, in load_weights
with h5py.File(filepath, 'r') as f:
File "/usr/local/lib/python3.7/dist-packages/h5py/_hl/files.py", line 427, in init
swmr=swmr)
File "/usr/local/lib/python3.7/dist-packages/h5py/_hl/files.py", line 190, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 96, in h5py.h5f.open
OSError: Unable to open file (file signature not found)

yolov3 is too slow

yolov3 you use is the reference zzh8829 author, deeo_sort is the reference Qidian213.. yolov3 is too slow and unfriendly to small targets

Error by executing tracker

When I execute the tracker on the webcam:
python object_tracker.py --video 0 --output ./data/video/results.avi

I receive following error. I already installed pkg-config and libgtk2.0-dev. Could you help me on what to do? Thank you.

File "/home/someone/miniconda3/envs/tracker-gpu/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "object_tracker.py", line 141, in main
cv2.imshow('output', img)
cv2.error: OpenCV(3.4.2) /tmp/build/80754af9/opencv-suite_1535558553474/work/modules/highgui/src/window.cpp:632: error: (-2:Unspecified error) The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Carbon support. If you are on Ubuntu or Debian, install libgtk2.0-dev and pkg-config, then re-run cmake or configure script in function 'cvShowImage'

error object_tracker.py Unable to open file

i have yolov3.weights but run object_tracker.py error ㅠㅠ

Traceback (most recent call last):
  File "object_tracker.py", line 168, in <module>
    app.run(main)
  File "/opt/conda/lib/python3.6/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/opt/conda/lib/python3.6/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "object_tracker.py", line 59, in main
    yolo.load_weights(FLAGS.weights)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 181, in load_weights
    return super(Model, self).load_weights(filepath, by_name)
  File "/opt/conda/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/network.py", line 1171, in load_weights
    with h5py.File(filepath, 'r') as f:
  File "/opt/conda/lib/python3.6/site-packages/h5py/_hl/files.py", line 394, in __init__
    swmr=swmr)
  File "/opt/conda/lib/python3.6/site-packages/h5py/_hl/files.py", line 170, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 85, in h5py.h5f.open
OSError: Unable to open file (file signature not found)

extraction of feature vector

in the code you are calling the method: def encoder(image, boxes)
which is getting you the box patch and than convert it to the features variable.
can you please explain what does the features var holds? as this is not the softmax result of the net.
thanks.

Error executing object_tracker.py

I get this error after following the steps. I did not manipulate the code and/or use custom YOLO. If anyone knows how to fix this...
I'm using Python3.7, Windows 10 and the CPU

Spelling Mistake in the README

Object Tracking using YOLOv3, Deep Sort and Tensorflow

This repository implements YOLOv3 and Deep SORT in order to perfrom real-time object tracking. Yolov3 is an algorithm that uses deep convolutional neural networks to perform object detection. We can feed these object detections into Deep SORT (Simple Online and Realtime Tracking with a Deep Association Metric) in order for a real-time object tracker to be created.

Getting started

Conda (Recommended)

# Tensorflow CPU
conda env create -f conda-cpu.yml
conda activate tracker-cpu

# Tensorflow GPU
conda env create -f conda-gpu.yml
conda activate tracker-gpu

Pip

# TensorFlow CPU
pip install -r requirements.txt

# TensorFlow GPU
pip install -r requirements-gpu.txt

Nvidia Driver (For GPU, if you haven't set it up already)

# Ubuntu 18.04
sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt install nvidia-driver-430
# Windows/Other
https://www.nvidia.com/Download/index.aspx

Downloading official pre-trained weights

For Linux: Let's download official yolov3 weights pre-trained on COCO dataset.

# yolov3
wget https://pjreddie.com/media/files/yolov3.weights -O weights/yolov3.weights

# yolov3-tiny
wget https://pjreddie.com/media/files/yolov3-tiny.weights -O weights/yolov3-tiny.weights

For Windows:
You can download the yolov3 weights by clicking here and yolov3-tiny here then save them to the weights folder.

Using Custom trained weights

Learn How To Train Custom YOLOV3 Weights Here: https://www.youtube.com/watch?v=zJDUhGL26iU

Add your custom weights file to weights folder and your custom .names file into data/labels folder.

Saving your yolov3 weights as a TensorFlow model.

Load the weights using load_weights.py script. This will convert the yolov3 weights into TensorFlow .tf model files!

# yolov3
python load_weights.py

# yolov3-tiny
python load_weights.py --weights ./weights/yolov3-tiny.weights --output ./weights/yolov3-tiny.tf --tiny

# yolov3-custom (add --tiny flag if your custom weights were trained for tiny model)
python load_weights.py --weights ./weights/<YOUR CUSTOM WEIGHTS FILE> --output ./weights/yolov3-custom.tf --num_classes <# CLASSES>

After executing one of the above lines, you should see proper .tf files in your weights folder. You are now ready to run object tracker.

Running the Object Tracker

Now you can run the object tracker for whichever model you have created, pre-trained, tiny, or custom.

# yolov3 on video
python object_tracker.py --video ./data/video/test.mp4 --output ./data/video/results.avi

#yolov3 on webcam 
python object_tracker.py --video 0 --output ./data/video/results.avi

#yolov3-tiny 
python object_tracker.py --video ./data/video/test.mp4 --output ./data/video/results.avi --weights ./weights/yolov3-tiny.tf --tiny

#yolov3-custom (add --tiny flag if your custom weights were trained for tiny model)
python object_tracker.py --video ./data/video/test.mp4 --output ./data/video/results.avi --weights ./weights/yolov3-custom.tf --num_classes <# CLASSES> --classes ./data/labels/<YOUR CUSTOM .names FILE>

The output flag saves your object tracker results as an avi file for you to watch back. It is not necessary to have the flag if you don't want to save the resulting video.

There is a test video uploaded in the data/video folder called test.mp4. If you followed all the steps properly with the pre-trained coco yolov3.weights model then when your run the object tracker wiht the first command above you should see the following.

Video Example

Webcam Example

This is a demo of running the object tracker using the above command for running the object tracker on your webcam.

Command Line Args Reference

load_weights.py:
  --output: path to output
    (default: './weights/yolov3.tf')
  --[no]tiny: yolov3 or yolov3-tiny
    (default: 'false')
  --weights: path to weights file
    (default: './weights/yolov3.weights')
  --num_classes: number of classes in the model
    (default: '80')
    (an integer)
    
object_tracker.py:
  --classes: path to classes file
    (default: './data/labels/coco.names')
  --video: path to input video (use 0 for webcam)
    (default: './data/video/test.mp4')
  --output: path to output video (remember to set right codec for given format. e.g. XVID for .avi)
    (default: None)
  --output_format: codec used in VideoWriter when saving video to file
    (default: 'XVID)
  --[no]tiny: yolov3 or yolov3-tiny
    (default: 'false')
  --weights: path to weights file
    (default: './weights/yolov3.tf')
  --num_classes: number of classes in the model
    (default: '80')
    (an integer)
  --yolo_max_boxes: maximum number of detections at one time
    (default: '100')
    (an integer)
  --yolo_iou_threshold: iou threshold for how close two boxes can be before they are detected as one box
    (default: 0.5)
    (a float)
  --yolo_score_threshold: score threshold for confidence level in detection for detection to count
    (default: 0.5)
    (a float)

Acknowledgments

• Yolov3 TensorFlow Amazing Implementation
• Deep SORT Repository
• Yolo v3 official paper

ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.

Whenever I try to run the load_weights.py file, I get this issue:
(tracker-cpu) C:\Users\deadp\yolov3_deepsort>python load_weights.py
Traceback (most recent call last):
File "C:\Users\deadp\anaconda3\envs\tracker-cpu\lib\site-packages\tensorflow_core\python\pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\Users\deadp\anaconda3\envs\tracker-cpu\lib\site-packages\tensorflow_core\python\pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\Users\deadp\anaconda3\envs\tracker-cpu\lib\site-packages\tensorflow_core\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "C:\Users\deadp\anaconda3\envs\tracker-cpu\lib\imp.py", line 243, in load_module
return load_dynamic(name, filename, file)
File "C:\Users\deadp\anaconda3\envs\tracker-cpu\lib\imp.py", line 343, in load_dynamic
return _load(spec)
ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "load_weights.py", line 4, in
from yolov3_tf2.models import YoloV3, YoloV3Tiny
File "C:\Users\deadp\yolov3_deepsort\yolov3_tf2\models.py", line 4, in
import tensorflow as tf
File "C:\Users\deadp\anaconda3\envs\tracker-cpu\lib\site-packages\tensorflow_init_.py", line 98, in
from tensorflow_core import *
File "C:\Users\deadp\anaconda3\envs\tracker-cpu\lib\site-packages\tensorflow_core_init_.py", line 40, in
from tensorflow.python.tools import module_util as module_util
File "C:\Users\deadp\anaconda3\envs\tracker-cpu\lib\site-packages\tensorflow_init.py", line 50, in getattr
module = self.load()
File "C:\Users\deadp\anaconda3\envs\tracker-cpu\lib\site-packages\tensorflow_init.py", line 44, in _load
module = importlib.import_module(self.name)
File "C:\Users\deadp\anaconda3\envs\tracker-cpu\lib\importlib_init.py", line 127, in import_module
return _bootstrap.gcd_import(name[level:], package, level)
File "C:\Users\deadp\anaconda3\envs\tracker-cpu\lib\site-packages\tensorflow_core\python_init.py", line 49, in
from tensorflow.python import pywrap_tensorflow
File "C:\Users\deadp\anaconda3\envs\tracker-cpu\lib\site-packages\tensorflow_core\python\pywrap_tensorflow.py", line 74, in
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "C:\Users\deadp\anaconda3\envs\tracker-cpu\lib\site-packages\tensorflow_core\python\pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\Users\deadp\anaconda3\envs\tracker-cpu\lib\site-packages\tensorflow_core\python\pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\Users\deadp\anaconda3\envs\tracker-cpu\lib\site-packages\tensorflow_core\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "C:\Users\deadp\anaconda3\envs\tracker-cpu\lib\imp.py", line 243, in load_module
return load_dynamic(name, filename, file)
File "C:\Users\deadp\anaconda3\envs\tracker-cpu\lib\imp.py", line 343, in load_dynamic
return _load(spec)
ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/errors

for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.

error while loading custom yolov3 weights

got this error while trying to load weights from a custom yolov3 model:

  File "/home/tes/workspace/yolov3_deepsort/yolov3_tf2/utils.py", line 70, in load_darknet_weights
    conv_shape).transpose([2, 3, 1, 0])
ValueError: cannot reshape array of size 19880 into shape (512,256,3,3)

AttributeError: type object 'h5py.h5.H5PYConfig' has no attribute '__reduce_cython__'

always error. which version didi you install? tensorflow-gpu, python, h5py , cpython
can you show out the conda list ? thank you!

(tracker-gpu) D:\yolov3_deepsort-master>python object_tracker.py --video 0 --output ./output/video_custom.avi --weights ./weights/yolov3-custom.tf --num_classes 11 --classes ./model_data/coco_custom.names
2020-09-13 17:56:28.603017: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cudart64_101.dll
Traceback (most recent call last):
  File "object_tracker.py", line 7, in <module>
    import tensorflow as tf
  File "D:\Programs\Anaconda3\envs\tracker-gpu\lib\site-packages\tensorflow\__init__.py", line 41, in <module>
    from tensorflow.python.tools import module_util as _module_util
  File "D:\Programs\Anaconda3\envs\tracker-gpu\lib\site-packages\tensorflow\python\__init__.py", line 84, in <module>
    from tensorflow.python import keras
  File "D:\Programs\Anaconda3\envs\tracker-gpu\lib\site-packages\tensorflow\python\keras\__init__.py", line 27, in <module>
    from tensorflow.python.keras import models
  File "D:\Programs\Anaconda3\envs\tracker-gpu\lib\site-packages\tensorflow\python\keras\models.py", line 26, in <module>
    from tensorflow.python.keras.engine import network
  File "D:\Programs\Anaconda3\envs\tracker-gpu\lib\site-packages\tensorflow\python\keras\engine\network.py", line 47, in <module>
    from tensorflow.python.keras.saving import hdf5_format
  File "D:\Programs\Anaconda3\envs\tracker-gpu\lib\site-packages\tensorflow\python\keras\saving\hdf5_format.py", line 41, in <module>
    import h5py
  File "D:\Programs\Anaconda3\envs\tracker-gpu\lib\site-packages\h5py\__init__.py", line 34, in <module>
    from . import version
  File "D:\Programs\Anaconda3\envs\tracker-gpu\lib\site-packages\h5py\version.py", line 17, in <module>
    from . import h5 as _h5
  File "h5py\h5.pyx", line 41, in init h5py.h5
AttributeError: type object 'h5py.h5.H5PYConfig' has no attribute '__reduce_cython__'

(tracker-gpu) D:\yolov3_deepsort-master>conda list
# packages in environment at D:\Programs\Anaconda3\envs\tracker-gpu:
#
# Name                    Version                   Build  Channel
absl-py                   0.10.0                   pypi_0    pypi
astor                     0.8.1                    pypi_0    pypi
astunparse                1.6.3                    pypi_0    pypi
blas                      1.0                         mkl
ca-certificates           2020.7.22                     0
cachetools                4.1.1                    pypi_0    pypi
certifi                   2020.6.20                pypi_0    pypi
chardet                   3.0.4                    pypi_0    pypi
cudatoolkit               10.1.243             h74a9793_0
cudnn                     7.6.5                cuda10.1_0
cycler                    0.10.0                   py37_0
cython                    0.29.21                  pypi_0    pypi
freetype                  2.10.2               hd328e21_0
gast                      0.2.2                    pypi_0    pypi
google-auth               1.21.1                   pypi_0    pypi
google-auth-oauthlib      0.4.1                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
grpcio                    1.32.0                   pypi_0    pypi
h5py                      2.10.0                   pypi_0    pypi
hdf5                      1.8.20               hac2f561_1
icc_rt                    2019.0.0             h0cc432a_1
icu                       58.2                 ha925a31_3
idna                      2.10                     pypi_0    pypi
importlib-metadata        1.7.0                    pypi_0    pypi
intel-openmp              2020.2                      254
jpeg                      9b                   hb83a4c4_2
keras-applications        1.0.8                    pypi_0    pypi
keras-preprocessing       1.1.2                    pypi_0    pypi
kiwisolver                1.2.0            py37h74a9793_0
libopencv                 3.4.2                h20b85fd_0
libpng                    1.6.37               h2a8f88b_0
libtiff                   4.1.0                h56a325e_1
lxml                      4.5.2                    pypi_0    pypi
lz4-c                     1.9.2                h62dcd97_1
markdown                  3.2.2                    pypi_0    pypi
matplotlib                3.2.2                         0
matplotlib-base           3.2.2            py37h64f37c6_0
mkl                       2020.2                      256
mkl-service               2.3.0            py37hb782905_0
mkl_fft                   1.1.0            py37h45dec08_0
mkl_random                1.1.1            py37h47e9c7a_0
numpy                     1.18.5                   pypi_0    pypi
numpy-base                1.19.1           py37ha3acd2a_0
oauthlib                  3.1.0                    pypi_0    pypi
opencv                    3.4.2            py37h40b0b35_0
openssl                   1.1.1g               he774522_1
opt-einsum                3.3.0                    pypi_0    pypi
pandas                    1.1.2                    pypi_0    pypi
pip                       20.2.2                   py37_0
protobuf                  3.13.0                   pypi_0    pypi
py-opencv                 3.4.2            py37hc319ecb_0
pyasn1                    0.4.8                    pypi_0    pypi
pyasn1-modules            0.2.8                    pypi_0    pypi
pyparsing                 2.4.7                      py_0
pyqt                      5.9.2            py37h6538335_2
python                    3.7.4                h5263a28_0
python-dateutil           2.8.1                      py_0
pytz                      2020.1                   pypi_0    pypi
qt                        5.9.7            vc14h73c81de_0
requests                  2.24.0                   pypi_0    pypi
requests-oauthlib         1.3.0                    pypi_0    pypi
rsa                       4.6                      pypi_0    pypi
scipy                     1.4.1                    pypi_0    pypi
seaborn                   0.11.0                   pypi_0    pypi
setuptools                50.3.0                   pypi_0    pypi
sip                       4.19.8           py37h6538335_0
six                       1.15.0                     py_0
sqlite                    3.33.0               h2a8f88b_0
tensorboard               2.1.1                    pypi_0    pypi
tensorboard-plugin-wit    1.7.0                    pypi_0    pypi
tensorflow-gpu            2.2.0                    pypi_0    pypi
tensorflow-gpu-estimator  2.2.0                    pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
tornado                   6.0.4            py37he774522_1
tqdm                      4.49.0                   pypi_0    pypi
urllib3                   1.25.10                  pypi_0    pypi
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.16.27012          hf0eaf9b_3
werkzeug                  1.0.1                    pypi_0    pypi
wheel                     0.35.1                   pypi_0    pypi
wincertstore              0.2                      py37_0
wrapt                     1.12.1                   pypi_0    pypi
xz                        5.2.5                h62dcd97_0
zipp                      3.1.0                    pypi_0    pypi
zlib                      1.2.11               h62dcd97_4
zstd                      1.4.5                h04227a9_0

generate_detection

how to Find distance between detected object and camera and put it near the class name?

KeyError: "The name 'net/images:0' refers to a Tensor which does not exist. The operation, 'net/images', does not exist in the graph.

Facing this error. I am running this in Google Colab
I have followed your video tutorial and copy-pasted as it is.

Error while executing load_weights.py

Traceback (most recent call last):
File "load_weights.py", line 4, in
from yolov3_tf2.models import YoloV3, YoloV3Tiny
File "/Users/sarjhana/Downloads/yolov3_deepsort-master/yolov3_tf2/models.py", line 4, in
import tensorflow as tf
File "/Users/sarjhana/dlenv/lib/python3.7/site-packages/tensorflow/init.py", line 98, in
from tensorflow_core import *
File "/Users/sarjhana/dlenv/lib/python3.7/site-packages/tensorflow_core/init.py", line 40, in
from tensorflow.python.tools import module_util as _module_util
File "", line 983, in _find_and_load
File "", line 959, in _find_and_load_unlocked
File "/Users/sarjhana/dlenv/lib/python3.7/site-packages/tensorflow/init.py", line 50, in getattr
module = self._load()
File "/Users/sarjhana/dlenv/lib/python3.7/site-packages/tensorflow/init.py", line 44, in _load
module = _importlib.import_module(self.name)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/Users/sarjhana/dlenv/lib/python3.7/site-packages/tensorflow_core/python/init.py", line 52, in
from tensorflow.core.framework.graph_pb2 import *
File "/Users/sarjhana/dlenv/lib/python3.7/site-packages/tensorflow_core/core/framework/graph_pb2.py", line 7, in
from google.protobuf import descriptor as _descriptor
File "/Users/sarjhana/dlenv/lib/python3.7/site-packages/google/protobuf/init.py", line 37, in
import('pkg_resources').declare_namespace(name)
File "/Users/sarjhana/dlenv/lib/python3.7/site-packages/pkg_resources/init.py", line 83, in
import('pkg_resources.extern.packaging.requirements')
File "/Users/sarjhana/dlenv/lib/python3.7/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 9, in
from pkg_resources.extern.pyparsing import stringStart, stringEnd, originalTextFor, ParseException
File "", line 983, in _find_and_load
File "", line 967, in _find_and_load_unlocked
File "", line 668, in _load_unlocked
File "", line 638, in _load_backward_compatible
File "/Users/sarjhana/dlenv/lib/python3.7/site-packages/pkg_resources/extern/init.py", line 43, in load_module
import(extant)
File "/Users/sarjhana/dlenv/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 4756, in
_escapedPunc = Word( _bslash, r"[]-*.$+^?()~ ", exact=2 ).setParseAction(lambda s,l,t:t[0][1])
File "/Users/sarjhana/dlenv/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 1284, in setParseAction
self.parseAction = list(map(_trim_arity, list(fns)))
File "/Users/sarjhana/dlenv/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 1066, in _trim_arity
this_line = extract_stack(limit=2)[-1]
File "/Users/sarjhana/dlenv/lib/python3.7/site-packages/pkg_resources/_vendor/pyparsing.py", line 1050, in extract_stack
frame_summary = traceback.extract_stack(limit=-offset+limit-1)[offset]
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/traceback.py", line 211, in extract_stack
stack = StackSummary.extract(walk_stack(f), limit=limit)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/traceback.py", line 363, in extract
f.line
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/traceback.py", line 285, in line
self._line = linecache.getline(self.filename, self.lineno).strip()
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/linecache.py", line 16, in getline
lines = getlines(filename, module_globals)
File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/linecache.py", line 48, in getlines
for mod in sys.modules.values():
RuntimeError: dictionary changed size during iteration

Yolov4-Tiny+DeepSort

Hi Guys @ALL

Is there anyone who have tried Yolov4-Tiny+DeepSort?

The speed of yolov3, Tiny-v3, yolov4 is too slow with DeepSort.

Or any other suggestions for a real-time tracker with Yolo?

I doubt whether the reid module worked right

By running demo,I find that in this project id switch problem is more serious than other same project.It looks that reid is invalidated.Have you noticed this？Thank you

Using yolov4 instead of v3

Recently the new version came out. Is there an easy way to integrate it with the code?

Deepsort tracking uses all the CPU cores

while running detection over a video, I see that my entire CPU memory is being used. I'm not able to run it on multiple threads as it leads to slowness.
Did anyone face this issue ?
Any help would be appreciated

Changing division for anchor boxes in models.py for different resolution

Hi there! In /yolov3_tf2/models.py there is division of 416 for the anchor boxes. I custom trained yolov3 with 512x512 resolution. So do I have to change the divison with 512 instead of 416?
code:
yolo_anchors = np.array([(10, 13), (16, 30), (33, 23), (30, 61), (62, 45),
(59, 119), (116, 90), (156, 198), (373, 326)],
np.float32) / 416
Edit: Also in object_tracker.py we should not forget to change the size to the detect images:
flags.DEFINE_integer('size',416, 'resize images to') #change size according to resolution.

Thanks.

Error when using object_tracker.py

Hi on my virtual environment with all dependencies required by requirements-gpu.txt when I run object_tracker.py I got this error

python object_tracker.py --video ./data/video/test.mp4 --output ./data/video/results.avi --weights ./weights/yolov3-tiny.tf --tiny
2020-05-12 03:11:30.555970: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-05-12 03:11:30.574436: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-05-12 03:11:30.575043: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: GeForce RTX 2060 SUPER computeCapability: 7.5
coreClock: 1.68GHz coreCount: 34 deviceMemorySize: 7.79GiB deviceMemoryBandwidth: 417.29GiB/s
2020-05-12 03:11:30.575142: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-10.2/lib64
2020-05-12 03:11:30.576358: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-05-12 03:11:30.577462: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-05-12 03:11:30.577653: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-05-12 03:11:30.578887: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-05-12 03:11:30.579652: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-05-12 03:11:30.582240: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-05-12 03:11:30.582269: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1598] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2020-05-12 03:11:30.583396: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-05-12 03:11:30.608803: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 3398735000 Hz
2020-05-12 03:11:30.609290: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f72d8000b60 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-05-12 03:11:30.609314: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-05-12 03:11:30.611123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-05-12 03:11:30.611140: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108]
Traceback (most recent call last):
File "object_tracker.py", line 163, in
app.run(main)
File "/home/servinfo/.virtualenvs/conda/lib/python3.8/site-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/home/servinfo/.virtualenvs/conda/lib/python3.8/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "object_tracker.py", line 41, in main
encoder = gdet.create_box_encoder(model_filename, batch_size=1)
File "/home/servinfo/yolov3_deepsort/tools/generate_detections.py", line 103, in create_box_encoder
image_encoder = ImageEncoder(model_filename, input_name, output_name)
File "/home/servinfo/yolov3_deepsort/tools/generate_detections.py", line 83, in init
self.input_var = tf.get_default_graph().get_tensor_by_name(
File "/home/servinfo/.virtualenvs/conda/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 3688, in get_tensor_by_name
return self.as_graph_element(name, allow_tensor=True, allow_operation=False)
File "/home/servinfo/.virtualenvs/conda/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 3512, in as_graph_element
return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
File "/home/servinfo/.virtualenvs/conda/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 3552, in _as_graph_element_locked
raise KeyError("The name %s refers to a Tensor which does not "
KeyError: "The name 'net/images:0' refers to a Tensor which does not exist. The operation, 'net/images', does not exist in the graph."
(tracker-gpu) (conda) servinfo@servinfo-desktop:~/yolov3_deepsort$

Can you help me?
BR

not suitable with yolov4

hi, I just curious to use yolov4 with deepsort instead of yolov3, the code is not suitable. can you specify which part need to be change?

KeyError: "The name 'net/images:0' refers to a Tensor which does not exist. The operation, 'net/images', does not exist in the graph."

getting this error

Is there any data produced or only visual?

Hi,

Is there any data being produced (or the ability to add count lines) to see the number of people counted in each direction? Or is this purely for resulting video display?

Issue in generating result.avi file - nothing inside .avi file

Hi man, I running "object_tracker.py" in Colab, it gives me this error: "cannot connect to X server" but generated a file named results.avi. I tried running in browser using commands
!ffmpeg -i ./data/video/traffic-results2.avi ./data/video/output.mp4

from IPython.display import HTML
from base64 import b64encode
mp4 = open('./data/video/output.mp4','rb').read()
data_url = "data:video/mp4;base64," + b64encode(mp4).decode()
HTML(""" <video controls> <source src="%s" type="video/mp4"> </video> """ % data_url)
But output is displaying nothing. I even tried downloading .avi and while playing it, it is showing file is corrupt.

Getting error in encoder = gdet.create_box_encoder(model_filename, batch_size=1)

Hi theAIGuysCode, I have executed the object_tracker.py with all video parameters changed to my own video. However, I am getting the errors below:
Traceback (most recent call last):
File "C:\Users\teezq\PycharmProjects\RTHDCS(FYP)\RTHDCS.py", line 57, in
encoder = gdet.create_box_encoder(model_filename, batch_size=1)
File "C:\Users\teezq\PycharmProjects\RTHDCS(FYP)\tools\generate_detections.py", line 100, in create_box_encoder
image_encoder = ImageEncoder(model_filename, input_name, output_name)
File "C:\Users\teezq\PycharmProjects\RTHDCS(FYP)\tools\generate_detections.py", line 80, in init
self.input_var = tf.get_default_graph().get_tensor_by_name(
File "C:\Users\teezq\PycharmProjects\RTHDCS(FYP)\venv\lib\site-packages\tensorflow\python\framework\ops.py", line 3931, in get_tensor_by_name
return self.as_graph_element(name, allow_tensor=True, allow_operation=False)
File "C:\Users\teezq\PycharmProjects\RTHDCS(FYP)\venv\lib\site-packages\tensorflow\python\framework\ops.py", line 3755, in as_graph_element
return self._as_graph_element_locked(obj, allow_tensor, allow_operation)
File "C:\Users\teezq\PycharmProjects\RTHDCS(FYP)\venv\lib\site-packages\tensorflow\python\framework\ops.py", line 3795, in _as_graph_element_locked
raise KeyError("The name %s refers to a Tensor which does not "
KeyError: "The name 'net/images:0' refers to a Tensor which does not exist. The operation, 'net/images', does not exist in the graph."

Does anyone got any hint what is causing this error.
I have ensure all the parameters are correct at the line: encoder = gdet.create_box_encoder(model_filename, batch_size=1). Moreover, I have redownloaded and replaced the "generate_detections.py" file multiple times.
Hoping to hear from your theAIGuysCode, I have really been working out on this error.

YoloV4

Hi I'd like to use your script with a YoloV4, but load_weights.py fails in converting the model in tf format.
Do you have any suggestion on how to make the script work for the YoloV4 as well?

importing issue with preprocess_image

error while running object_tracker.py

While running on custom data i get this error

W0924 14:33:41.166774 2940 object_tracker.py:83] Empty Frame
W0924 14:33:41.268894 2940 object_tracker.py:83] Empty Frame
W0924 14:33:41.372303 2940 object_tracker.py:83] Empty Frame

Installation of nvidia driver

it should be:
sudo add-apt-repository ppa:graphics-drivers/ppa

instead of:
sudo apt-add-repository -r ppa:graphics-drivers/ppa

i changed line 83 and 85 as mentioned before but i still get the problem:code is in pycharm!

when load h5 file:ValueError: You are trying to load a weight file containing 45 layers into a model with 7 layers.

how to solve~

Error when executing object tracker: tensorflow.python.framework.errors_impl.ResourceExhaustedError

Hi,
when I tried to run the object tracker with
python object_tracker.py --video 0 --output ./data/video/results.avi

unfortunatelly an error occurs. You can find the whole output below.

Here are some of the errors which are listed within the complete output below:

2020-09-15 16:27:45.302542: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
...
2020-09-15 16:28:22.690033: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at cwise_ops_common.cc:82 : Resource exhausted: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
...
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add] name: conv2d_49/kernel/Initializer/random_uniform/

I'm using a Jetson Nano with JetPack 4.3 and Tensorflow version 2.1.0.

I would appreciate some good advice.
BR

Complete output text

2020-09-15 16:27:28.567398: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-15 16:27:33.539154: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer.so.6
2020-09-15 16:27:33.571472: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libnvinfer_plugin.so.6
2020-09-15 16:27:44.860933: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-09-15 16:27:44.902688: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:44.902874: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 23.84GiB/s
2020-09-15 16:27:44.903030: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-15 16:27:44.903205: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-09-15 16:27:44.984059: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-09-15 16:27:45.093582: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-09-15 16:27:45.206979: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-09-15 16:27:45.270792: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-09-15 16:27:45.271229: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-15 16:27:45.271733: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:45.272165: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:45.272330: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-09-15 16:27:45.302542: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2020-09-15 16:27:45.303124: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x11c62ec0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-09-15 16:27:45.303181: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-09-15 16:27:45.405645: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:45.405961: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x11cb26c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-09-15 16:27:45.406032: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2020-09-15 16:27:45.406766: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:45.406916: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 23.84GiB/s
2020-09-15 16:27:45.407163: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-15 16:27:45.407278: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-09-15 16:27:45.407412: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-09-15 16:27:45.407530: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-09-15 16:27:45.407637: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-09-15 16:27:45.407744: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-09-15 16:27:45.407832: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-15 16:27:45.408170: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:45.408587: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:45.408671: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-09-15 16:27:45.408832: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-15 16:27:55.785902: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-15 16:27:55.786029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0
2020-09-15 16:27:55.786074: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N
2020-09-15 16:27:55.814766: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:55.815485: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:55.815804: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 183 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-09-15 16:27:57.130111: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:57.146867: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: NVIDIA Tegra X1 computeCapability: 5.3
coreClock: 0.9216GHz coreCount: 1 deviceMemorySize: 3.87GiB deviceMemoryBandwidth: 23.84GiB/s
2020-09-15 16:27:57.162859: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-09-15 16:27:57.194069: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-09-15 16:27:57.311603: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-09-15 16:27:57.312279: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-09-15 16:27:57.322676: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-09-15 16:27:57.323023: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-09-15 16:27:57.357181: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-09-15 16:27:57.357784: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:57.358266: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:57.358402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-09-15 16:27:57.358507: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-09-15 16:27:57.358562: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0
2020-09-15 16:27:57.358607: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N
2020-09-15 16:27:57.359069: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:57.359611: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-09-15 16:27:57.359806: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 183 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
2020-09-15 16:28:22.589148: W tensorflow/core/common_runtime/bfc_allocator.cc:424] Allocator (GPU_0_bfc) ran out of memory trying to allocate 18.00MiB (rounded to 18874368). Current allocation summary follows.
2020-09-15 16:28:22.603518: I tensorflow/core/common_runtime/bfc_allocator.cc:894] BFCAllocator dump for GPU_0_bfc
2020-09-15 16:28:22.603676: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (256): Total Chunks: 25, Chunks in use: 25. 6.2KiB allocated for chunks. 6.2KiB in use in bin. 5.0KiB client-requested in use in bin.
2020-09-15 16:28:22.603727: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (512): Total Chunks: 44, Chunks in use: 44. 22.0KiB allocated for chunks. 22.0KiB in use in bin. 22.0KiB client-requested in use in bin.
2020-09-15 16:28:22.603769: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1024): Total Chunks: 69, Chunks in use: 69. 69.2KiB allocated for chunks. 69.2KiB in use in bin. 69.0KiB client-requested in use in bin.
2020-09-15 16:28:22.603811: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2048): Total Chunks: 49, Chunks in use: 49. 101.0KiB allocated for chunks. 101.0KiB in use in bin. 99.4KiB client-requested in use in bin.
2020-09-15 16:28:22.603977: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4096): Total Chunks: 12, Chunks in use: 12. 48.0KiB allocated for chunks. 48.0KiB in use in bin. 48.0KiB client-requested in use in bin.
2020-09-15 16:28:22.604033: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8192): Total Chunks: 1, Chunks in use: 1. 8.0KiB allocated for chunks. 8.0KiB in use in bin. 8.0KiB client-requested in use in bin.
2020-09-15 16:28:22.604092: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16384): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-15 16:28:22.604165: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (32768): Total Chunks: 3, Chunks in use: 2. 102.0KiB allocated for chunks. 64.0KiB in use in bin. 64.0KiB client-requested in use in bin.
2020-09-15 16:28:22.604297: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (65536): Total Chunks: 2, Chunks in use: 2. 144.0KiB allocated for chunks. 144.0KiB in use in bin. 144.0KiB client-requested in use in bin.
2020-09-15 16:28:22.604367: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (131072): Total Chunks: 9, Chunks in use: 8. 1.14MiB allocated for chunks. 1.00MiB in use in bin. 1.00MiB client-requested in use in bin.
2020-09-15 16:28:22.604435: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (262144): Total Chunks: 3, Chunks in use: 3. 864.0KiB allocated for chunks. 864.0KiB in use in bin. 864.0KiB client-requested in use in bin.
2020-09-15 16:28:22.604496: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (524288): Total Chunks: 10, Chunks in use: 8. 5.28MiB allocated for chunks. 4.28MiB in use in bin. 4.00MiB client-requested in use in bin.
2020-09-15 16:28:22.604655: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (1048576): Total Chunks: 9, Chunks in use: 9. 10.88MiB allocated for chunks. 10.88MiB in use in bin. 10.12MiB client-requested in use in bin.
2020-09-15 16:28:22.604720: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (2097152): Total Chunks: 5, Chunks in use: 3. 11.00MiB allocated for chunks. 6.50MiB in use in bin. 6.00MiB client-requested in use in bin.
2020-09-15 16:28:22.604786: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (4194304): Total Chunks: 10, Chunks in use: 9. 46.88MiB allocated for chunks. 42.38MiB in use in bin. 40.50MiB client-requested in use in bin.
2020-09-15 16:28:22.604846: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-15 16:28:22.604909: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (16777216): Total Chunks: 4, Chunks in use: 4. 74.35MiB allocated for chunks. 74.35MiB in use in bin. 72.00MiB client-requested in use in bin.
2020-09-15 16:28:22.604972: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (33554432): Total Chunks: 1, Chunks in use: 1. 32.50MiB allocated for chunks. 32.50MiB in use in bin. 18.00MiB client-requested in use in bin.
2020-09-15 16:28:22.605121: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-15 16:28:22.605181: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-15 16:28:22.605238: I tensorflow/core/common_runtime/bfc_allocator.cc:901] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2020-09-15 16:28:22.605298: I tensorflow/core/common_runtime/bfc_allocator.cc:917] Bin for 18.00MiB was 16.00MiB, Chunk State:
2020-09-15 16:28:22.605339: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 1048576
2020-09-15 16:28:22.647695: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0000 of size 1280 next 1
2020-09-15 16:28:22.647782: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0500 of size 256 next 2
2020-09-15 16:28:22.647942: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0600 of size 256 next 5
2020-09-15 16:28:22.647989: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0700 of size 256 next 4
2020-09-15 16:28:22.648030: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0800 of size 256 next 7
2020-09-15 16:28:22.648073: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0900 of size 256 next 8
2020-09-15 16:28:22.648110: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0a00 of size 256 next 9
2020-09-15 16:28:22.648155: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0b00 of size 256 next 11
2020-09-15 16:28:22.648260: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0c00 of size 256 next 13
2020-09-15 16:28:22.648326: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0d00 of size 256 next 14
2020-09-15 16:28:22.648377: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0e00 of size 256 next 15
2020-09-15 16:28:22.648426: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f0f00 of size 256 next 17
2020-09-15 16:28:22.648589: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1000 of size 256 next 19
2020-09-15 16:28:22.648640: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1100 of size 256 next 20
2020-09-15 16:28:22.648681: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1200 of size 256 next 22
2020-09-15 16:28:22.648734: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1300 of size 256 next 24
2020-09-15 16:28:22.648776: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1400 of size 256 next 25
2020-09-15 16:28:22.648820: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1500 of size 256 next 26
2020-09-15 16:28:22.648865: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1600 of size 512 next 27
2020-09-15 16:28:22.648910: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1800 of size 512 next 30
2020-09-15 16:28:22.648954: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1a00 of size 512 next 31
2020-09-15 16:28:22.648993: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1c00 of size 512 next 32
2020-09-15 16:28:22.649031: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1e00 of size 256 next 33
2020-09-15 16:28:22.649068: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f1f00 of size 256 next 35
2020-09-15 16:28:22.649225: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2000 of size 256 next 36
2020-09-15 16:28:22.649269: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2100 of size 256 next 37
2020-09-15 16:28:22.649308: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2200 of size 512 next 3
2020-09-15 16:28:22.649349: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f2400 of size 3584 next 6
2020-09-15 16:28:22.649388: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3200 of size 512 next 38
2020-09-15 16:28:22.649428: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3400 of size 512 next 40
2020-09-15 16:28:22.649515: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3600 of size 512 next 41
2020-09-15 16:28:22.649556: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3800 of size 256 next 42
2020-09-15 16:28:22.649601: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3900 of size 256 next 44
2020-09-15 16:28:22.649640: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3a00 of size 256 next 45
2020-09-15 16:28:22.649691: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3b00 of size 256 next 46
2020-09-15 16:28:22.649728: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3c00 of size 512 next 47
2020-09-15 16:28:22.649770: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f3e00 of size 512 next 49
2020-09-15 16:28:22.649811: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4000 of size 512 next 50
2020-09-15 16:28:22.649857: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4200 of size 512 next 51
2020-09-15 16:28:22.649906: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4400 of size 1024 next 53
2020-09-15 16:28:22.649947: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4800 of size 1024 next 56
2020-09-15 16:28:22.649992: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f4c00 of size 1024 next 57
2020-09-15 16:28:22.650032: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5000 of size 1024 next 58
2020-09-15 16:28:22.650076: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5400 of size 512 next 59
2020-09-15 16:28:22.650115: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5600 of size 512 next 61
2020-09-15 16:28:22.650165: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5800 of size 512 next 63
2020-09-15 16:28:22.650308: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5a00 of size 512 next 64
2020-09-15 16:28:22.650350: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f5c00 of size 1024 next 65
2020-09-15 16:28:22.650392: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6000 of size 1024 next 66
2020-09-15 16:28:22.650430: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6400 of size 1024 next 67
2020-09-15 16:28:22.651437: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6800 of size 1024 next 68
2020-09-15 16:28:22.651528: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6c00 of size 512 next 69
2020-09-15 16:28:22.651578: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f6e00 of size 512 next 71
2020-09-15 16:28:22.651621: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f7000 of size 512 next 18
2020-09-15 16:28:22.651668: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f7200 of size 8192 next 16
2020-09-15 16:28:22.651713: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f008f9200 of size 32768 next 34
2020-09-15 16:28:22.651761: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00901200 of size 32768 next 43
2020-09-15 16:28:22.651805: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00909200 of size 512 next 72
2020-09-15 16:28:22.651847: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00909400 of size 1024 next 75
2020-09-15 16:28:22.651894: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00909800 of size 1024 next 76
2020-09-15 16:28:22.651943: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00909c00 of size 1024 next 77
2020-09-15 16:28:22.651986: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090a000 of size 1024 next 78
2020-09-15 16:28:22.652030: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090a400 of size 512 next 79
2020-09-15 16:28:22.652070: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090a600 of size 512 next 80
2020-09-15 16:28:22.652311: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090a800 of size 512 next 82
2020-09-15 16:28:22.652408: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090aa00 of size 512 next 83
2020-09-15 16:28:22.652459: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090ac00 of size 1024 next 85
2020-09-15 16:28:22.652500: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090b000 of size 1024 next 86
2020-09-15 16:28:22.652543: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090b400 of size 1024 next 87
2020-09-15 16:28:22.652589: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090b800 of size 1024 next 88
2020-09-15 16:28:22.652870: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090bc00 of size 512 next 90
2020-09-15 16:28:22.652933: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090be00 of size 512 next 91
2020-09-15 16:28:22.652982: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090c000 of size 512 next 92
2020-09-15 16:28:22.653025: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090c200 of size 512 next 93
2020-09-15 16:28:22.653055: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090c400 of size 1024 next 95
2020-09-15 16:28:22.653081: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090c800 of size 1024 next 97
2020-09-15 16:28:22.653107: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090cc00 of size 1024 next 98
2020-09-15 16:28:22.653130: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090d000 of size 1024 next 99
2020-09-15 16:28:22.653160: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090d400 of size 512 next 101
2020-09-15 16:28:22.653842: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090d600 of size 512 next 102
2020-09-15 16:28:22.657624: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090d800 of size 512 next 103
2020-09-15 16:28:22.657723: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090da00 of size 512 next 104
2020-09-15 16:28:22.657783: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090dc00 of size 1024 next 106
2020-09-15 16:28:22.657836: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090e000 of size 1024 next 107
2020-09-15 16:28:22.657896: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090e400 of size 1024 next 108
2020-09-15 16:28:22.657941: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090e800 of size 1024 next 109
2020-09-15 16:28:22.657984: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090ec00 of size 512 next 111
2020-09-15 16:28:22.658032: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090ee00 of size 512 next 112
2020-09-15 16:28:22.658089: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090f000 of size 512 next 113
2020-09-15 16:28:22.658132: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090f200 of size 512 next 114
2020-09-15 16:28:22.658174: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090f400 of size 1024 next 116
2020-09-15 16:28:22.658213: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090f800 of size 1024 next 117
2020-09-15 16:28:22.658256: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0090fc00 of size 1024 next 118
2020-09-15 16:28:22.658295: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910000 of size 1024 next 119
2020-09-15 16:28:22.658334: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910400 of size 512 next 121
2020-09-15 16:28:22.658373: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910600 of size 512 next 122
2020-09-15 16:28:22.658411: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910800 of size 512 next 123
2020-09-15 16:28:22.658449: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910a00 of size 512 next 124
2020-09-15 16:28:22.658489: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00910c00 of size 1024 next 125
2020-09-15 16:28:22.658527: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911000 of size 1024 next 126
2020-09-15 16:28:22.658567: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911400 of size 1024 next 127
2020-09-15 16:28:22.658605: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911800 of size 1024 next 128
2020-09-15 16:28:22.658644: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911c00 of size 512 next 130
2020-09-15 16:28:22.658682: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00911e00 of size 512 next 131
2020-09-15 16:28:22.658720: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912000 of size 512 next 132
2020-09-15 16:28:22.658758: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912200 of size 512 next 133
2020-09-15 16:28:22.658798: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912400 of size 1024 next 136
2020-09-15 16:28:22.658836: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912800 of size 1024 next 137
2020-09-15 16:28:22.658874: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00912c00 of size 1024 next 138
2020-09-15 16:28:22.658913: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00913000 of size 1024 next 139
2020-09-15 16:28:22.658957: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00913400 of size 2048 next 140
2020-09-15 16:28:22.659005: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00913c00 of size 2048 next 141
2020-09-15 16:28:22.659055: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00914400 of size 2048 next 142
2020-09-15 16:28:22.659102: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00914c00 of size 2048 next 144
2020-09-15 16:28:22.659150: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00915400 of size 1024 next 145
2020-09-15 16:28:22.659197: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00915800 of size 1024 next 146
2020-09-15 16:28:22.659246: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00915c00 of size 1024 next 148
2020-09-15 16:28:22.659301: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00916000 of size 1024 next 149
2020-09-15 16:28:22.659351: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00916400 of size 3584 next 12
2020-09-15 16:28:22.659394: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00917200 of size 73728 next 10
2020-09-15 16:28:22.659445: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00929200 of size 2048 next 150
2020-09-15 16:28:22.659698: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00929a00 of size 2048 next 153
2020-09-15 16:28:22.659763: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092a200 of size 2048 next 154
2020-09-15 16:28:22.659810: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092aa00 of size 1024 next 156
2020-09-15 16:28:22.659851: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092ae00 of size 1024 next 157
2020-09-15 16:28:22.659891: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092b200 of size 1024 next 158
2020-09-15 16:28:22.659937: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092b600 of size 1024 next 159
2020-09-15 16:28:22.659981: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092ba00 of size 2048 next 160
2020-09-15 16:28:22.660023: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092c200 of size 2048 next 162
2020-09-15 16:28:22.660065: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092ca00 of size 2048 next 163
2020-09-15 16:28:22.660107: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092d200 of size 2048 next 164
2020-09-15 16:28:22.660151: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092da00 of size 1024 next 165
2020-09-15 16:28:22.660233: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092de00 of size 1024 next 166
2020-09-15 16:28:22.660283: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092e200 of size 1024 next 167
2020-09-15 16:28:22.660327: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092e600 of size 1024 next 168
2020-09-15 16:28:22.660374: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092ea00 of size 2048 next 169
2020-09-15 16:28:22.660419: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092f200 of size 2048 next 171
2020-09-15 16:28:22.660462: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0092fa00 of size 2048 next 172
2020-09-15 16:28:22.660502: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00930200 of size 2048 next 173
2020-09-15 16:28:22.660540: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00930a00 of size 1024 next 175
2020-09-15 16:28:22.660581: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00930e00 of size 1024 next 176
2020-09-15 16:28:22.660622: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00931200 of size 1024 next 177
2020-09-15 16:28:22.660661: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00931600 of size 1024 next 178
2020-09-15 16:28:22.660699: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00931a00 of size 2048 next 180
2020-09-15 16:28:22.660738: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00932200 of size 2048 next 182
2020-09-15 16:28:22.660787: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00932a00 of size 2048 next 183
2020-09-15 16:28:22.660841: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00933200 of size 2048 next 184
2020-09-15 16:28:22.660885: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00933a00 of size 1024 next 186
2020-09-15 16:28:22.660929: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00933e00 of size 1024 next 187
2020-09-15 16:28:22.661114: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00934200 of size 1024 next 188
2020-09-15 16:28:22.661160: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00934600 of size 1024 next 189
2020-09-15 16:28:22.661207: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00934a00 of size 2048 next 190
2020-09-15 16:28:22.661248: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00935200 of size 2048 next 191
2020-09-15 16:28:22.661290: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00935a00 of size 2048 next 193
2020-09-15 16:28:22.661332: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00936200 of size 2048 next 194
2020-09-15 16:28:22.661377: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00936a00 of size 1024 next 196
2020-09-15 16:28:22.661420: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00936e00 of size 1024 next 197
2020-09-15 16:28:22.661467: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00937200 of size 1024 next 198
2020-09-15 16:28:22.661512: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00937600 of size 1024 next 199
2020-09-15 16:28:22.661561: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00937a00 of size 2048 next 202
2020-09-15 16:28:22.661610: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00938200 of size 2048 next 203
2020-09-15 16:28:22.661655: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00938a00 of size 2048 next 204
2020-09-15 16:28:22.661695: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00939200 of size 2048 next 205
2020-09-15 16:28:22.661756: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00939a00 of size 1024 next 207
2020-09-15 16:28:22.661805: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00939e00 of size 1024 next 208
2020-09-15 16:28:22.661849: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0093a200 of size 1024 next 209
2020-09-15 16:28:22.661897: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0093a600 of size 1024 next 210
2020-09-15 16:28:22.661944: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0093aa00 of size 2048 next 23
2020-09-15 16:28:22.662090: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0093b200 of size 73728 next 21
2020-09-15 16:28:22.662145: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094d200 of size 2048 next 212
2020-09-15 16:28:22.662195: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094da00 of size 2048 next 213
2020-09-15 16:28:22.662245: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094e200 of size 2048 next 214
2020-09-15 16:28:22.662296: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094ea00 of size 1024 next 216
2020-09-15 16:28:22.662346: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094ee00 of size 1024 next 217
2020-09-15 16:28:22.662392: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094f200 of size 1024 next 218
2020-09-15 16:28:22.662440: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094f600 of size 1024 next 219
2020-09-15 16:28:22.662486: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0094fa00 of size 2048 next 221
2020-09-15 16:28:22.662530: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00950200 of size 2048 next 222
2020-09-15 16:28:22.662573: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00950a00 of size 2048 next 223
2020-09-15 16:28:22.662721: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00951200 of size 2048 next 224
2020-09-15 16:28:22.662773: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00951a00 of size 4096 next 225
2020-09-15 16:28:22.662818: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00952a00 of size 4096 next 228
2020-09-15 16:28:22.662861: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00953a00 of size 4096 next 229
2020-09-15 16:28:22.662908: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00954a00 of size 4096 next 230
2020-09-15 16:28:22.662952: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00955a00 of size 2048 next 231
2020-09-15 16:28:22.662999: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00956200 of size 2048 next 232
2020-09-15 16:28:22.663046: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00956a00 of size 2048 next 234
2020-09-15 16:28:22.663093: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00957200 of size 2048 next 235
2020-09-15 16:28:22.663141: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00957a00 of size 4096 next 236
2020-09-15 16:28:22.663185: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00958a00 of size 4096 next 238
2020-09-15 16:28:22.663226: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00959a00 of size 4096 next 239
2020-09-15 16:28:22.663267: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095aa00 of size 4096 next 240
2020-09-15 16:28:22.663406: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095ba00 of size 2048 next 242
2020-09-15 16:28:22.663517: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095c200 of size 2048 next 243
2020-09-15 16:28:22.664124: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095ca00 of size 2048 next 244
2020-09-15 16:28:22.664229: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095d200 of size 2048 next 245
2020-09-15 16:28:22.664280: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095da00 of size 4096 next 246
2020-09-15 16:28:22.664324: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095ea00 of size 4096 next 247
2020-09-15 16:28:22.664365: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0095fa00 of size 4096 next 249
2020-09-15 16:28:22.664406: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00960a00 of size 4096 next 250
2020-09-15 16:28:22.664445: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00961a00 of size 2048 next 252
2020-09-15 16:28:22.664801: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00962200 of size 2048 next 253
2020-09-15 16:28:22.664860: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00962a00 of size 2048 next 254
2020-09-15 16:28:22.664901: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00963200 of size 2048 next 255
2020-09-15 16:28:22.666022: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f00963a00 of size 38912 next 81
2020-09-15 16:28:22.666098: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0096d200 of size 131072 next 62
2020-09-15 16:28:22.666143: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0098d200 of size 131072 next 60
2020-09-15 16:28:22.666186: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009ad200 of size 131072 next 70
2020-09-15 16:28:22.666227: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f009cd200 of size 142848 next 18446744073709551615
2020-09-15 16:28:22.666269: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 2097152
2020-09-15 16:28:22.666313: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f009f0000 of size 294912 next 29
2020-09-15 16:28:22.666354: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00a38000 of size 294912 next 39
2020-09-15 16:28:22.666392: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00a80000 of size 294912 next 48
2020-09-15 16:28:22.666431: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00ac8000 of size 524288 next 155
2020-09-15 16:28:22.666470: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00b48000 of size 688128 next 18446744073709551615
2020-09-15 16:28:22.666507: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 4194304
2020-09-15 16:28:22.666545: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00bf0000 of size 131072 next 89
2020-09-15 16:28:22.666584: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00c10000 of size 131072 next 100
2020-09-15 16:28:22.666622: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00c30000 of size 131072 next 110
2020-09-15 16:28:22.666663: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00c50000 of size 131072 next 120
2020-09-15 16:28:22.666703: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00c70000 of size 131072 next 129
2020-09-15 16:28:22.666744: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f00c90000 of size 524288 next 55
2020-09-15 16:28:22.666785: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00d10000 of size 1179648 next 52
2020-09-15 16:28:22.666830: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00e30000 of size 1835008 next 18446744073709551615
2020-09-15 16:28:22.666876: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 8388608
2020-09-15 16:28:22.666921: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f00ff0000 of size 1179648 next 74
2020-09-15 16:28:22.666969: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01110000 of size 1179648 next 84
2020-09-15 16:28:22.667018: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f01230000 of size 524288 next 147
2020-09-15 16:28:22.667060: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f012b0000 of size 655360 next 96
2020-09-15 16:28:22.667510: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01350000 of size 1179648 next 94
2020-09-15 16:28:22.667567: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01470000 of size 1179648 next 105
2020-09-15 16:28:22.667607: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01590000 of size 1179648 next 115
2020-09-15 16:28:22.667646: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f016b0000 of size 1310720 next 18446744073709551615
2020-09-15 16:28:22.667687: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 16777216
2020-09-15 16:28:22.667727: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f017f0000 of size 1179648 next 135
2020-09-15 16:28:22.667766: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01910000 of size 524288 next 174
2020-09-15 16:28:22.667806: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01990000 of size 2097152 next 241
2020-09-15 16:28:22.669192: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f01b90000 of size 2097152 next 251
2020-09-15 16:28:22.669273: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f01d90000 of size 4718592 next 143
2020-09-15 16:28:22.669322: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f02210000 of size 6160384 next 18446744073709551615
2020-09-15 16:28:22.669366: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 33554432
2020-09-15 16:28:22.669411: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f027f0000 of size 4718592 next 152
2020-09-15 16:28:22.669454: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f02c70000 of size 4718592 next 161
2020-09-15 16:28:22.669494: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f030f0000 of size 4718592 next 170
2020-09-15 16:28:22.669533: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f03570000 of size 524288 next 185
2020-09-15 16:28:22.669573: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f035f0000 of size 524288 next 195
2020-09-15 16:28:22.669614: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f03670000 of size 524288 next 206
2020-09-15 16:28:22.669658: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f036f0000 of size 524288 next 215
2020-09-15 16:28:22.669700: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f03770000 of size 2621440 next 181
2020-09-15 16:28:22.669744: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f039f0000 of size 4718592 next 179
2020-09-15 16:28:22.669790: I tensorflow/core/common_runtime/bfc_allocator.cc:950] Free at f03e70000 of size 2097152 next 233
2020-09-15 16:28:22.669831: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f04070000 of size 2621440 next 192
2020-09-15 16:28:22.669872: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f042f0000 of size 5242880 next 18446744073709551615
2020-09-15 16:28:22.669913: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 67108864
2020-09-15 16:28:22.669955: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f047f0000 of size 4718592 next 201
2020-09-15 16:28:22.669996: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f04c70000 of size 4718592 next 211
2020-09-15 16:28:22.670131: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f050f0000 of size 4718592 next 220
2020-09-15 16:28:22.670174: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f05570000 of size 18874368 next 237
2020-09-15 16:28:22.670213: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f06770000 of size 34078720 next 18446744073709551615
2020-09-15 16:28:22.670249: I tensorflow/core/common_runtime/bfc_allocator.cc:930] Next region of size 59088896
2020-09-15 16:28:22.670290: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f087f0000 of size 18874368 next 227
2020-09-15 16:28:22.670328: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f099f0000 of size 18874368 next 248
2020-09-15 16:28:22.670366: I tensorflow/core/common_runtime/bfc_allocator.cc:950] InUse at f0abf0000 of size 21340160 next 18446744073709551615
2020-09-15 16:28:22.670401: I tensorflow/core/common_runtime/bfc_allocator.cc:955] Summary of in-use Chunks by size:
2020-09-15 16:28:22.670449: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 25 Chunks of size 256 totalling 6.2KiB
2020-09-15 16:28:22.670494: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 44 Chunks of size 512 totalling 22.0KiB
2020-09-15 16:28:22.670537: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 68 Chunks of size 1024 totalling 68.0KiB
2020-09-15 16:28:22.670577: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 1280 totalling 1.2KiB
2020-09-15 16:28:22.670619: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 47 Chunks of size 2048 totalling 94.0KiB
2020-09-15 16:28:22.670798: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 3584 totalling 7.0KiB
2020-09-15 16:28:22.671463: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 12 Chunks of size 4096 totalling 48.0KiB
2020-09-15 16:28:22.671519: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 8192 totalling 8.0KiB
2020-09-15 16:28:22.671553: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 32768 totalling 64.0KiB
2020-09-15 16:28:22.671587: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 73728 totalling 144.0KiB
2020-09-15 16:28:22.671621: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 8 Chunks of size 131072 totalling 1.00MiB
2020-09-15 16:28:22.671654: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 3 Chunks of size 294912 totalling 864.0KiB
2020-09-15 16:28:22.671686: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 6 Chunks of size 524288 totalling 3.00MiB
2020-09-15 16:28:22.671723: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 655360 totalling 640.0KiB
2020-09-15 16:28:22.671762: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 688128 totalling 672.0KiB
2020-09-15 16:28:22.671795: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 7 Chunks of size 1179648 totalling 7.88MiB
2020-09-15 16:28:22.671828: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 1310720 totalling 1.25MiB
2020-09-15 16:28:22.671862: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 1835008 totalling 1.75MiB
2020-09-15 16:28:22.671895: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 2 Chunks of size 2097152 totalling 4.00MiB
2020-09-15 16:28:22.671924: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 2621440 totalling 2.50MiB
2020-09-15 16:28:22.671953: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 7 Chunks of size 4718592 totalling 31.50MiB
2020-09-15 16:28:22.671982: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 5242880 totalling 5.00MiB
2020-09-15 16:28:22.672012: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 6160384 totalling 5.88MiB
2020-09-15 16:28:22.672040: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 3 Chunks of size 18874368 totalling 54.00MiB
2020-09-15 16:28:22.672068: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 21340160 totalling 20.35MiB
2020-09-15 16:28:22.672097: I tensorflow/core/common_runtime/bfc_allocator.cc:958] 1 Chunks of size 34078720 totalling 32.50MiB
2020-09-15 16:28:22.672126: I tensorflow/core/common_runtime/bfc_allocator.cc:962] Sum Total of in-use chunks: 173.18MiB
2020-09-15 16:28:22.672152: I tensorflow/core/common_runtime/bfc_allocator.cc:964] total_region_allocated_bytes_: 192258048 memory_limit_: 192258048 available bytes: 0 curr_region_allocation_bytes_: 268435456
2020-09-15 16:28:22.672265: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Stats:
Limit: 192258048
InUse: 181590528
MaxInUse: 181590528
NumAllocs: 1232
MaxAllocSize: 34078720

2020-09-15 16:28:22.672343: W tensorflow/core/common_runtime/bfc_allocator.cc:429] ************_*******************xxxxxxxx
2020-09-15 16:28:22.690033: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at cwise_ops_common.cc:82 : Resource exhausted: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "object_tracker.py", line 163, in
app.run(main)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "object_tracker.py", line 52, in main
yolo = YoloV3(classes=FLAGS.num_classes)
File "/home/christopher/aiguyyolotest2/yolov3_deepsort/yolov3_tf2/models.py", line 211, in YoloV3
x_36, x_61, x = Darknet(name='yolo_darknet')(x)
File "/home/christopher/aiguyyolotest2/yolov3_deepsort/yolov3_tf2/models.py", line 78, in Darknet
x = DarknetBlock(x, 1024, 4)
File "/home/christopher/aiguyyolotest2/yolov3_deepsort/yolov3_tf2/models.py", line 67, in DarknetBlock
x = DarknetResidual(x, filters)
File "/home/christopher/aiguyyolotest2/yolov3_deepsort/yolov3_tf2/models.py", line 59, in DarknetResidual
x = DarknetConv(x, filters, 3)
File "/home/christopher/aiguyyolotest2/yolov3_deepsort/yolov3_tf2/models.py", line 49, in DarknetConv
use_bias=not batch_norm, kernel_regularizer=l2(0.0005))(x)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 748, in call
self._maybe_build(inputs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 2116, in _maybe_build
self.build(input_shapes)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/layers/convolutional.py", line 161, in build
dtype=self.dtype)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 446, in add_weight
caching_device=caching_device)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/base.py", line 744, in _add_variable_with_custom_getter
**kwargs_for_getter)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py", line 142, in make_variable
shape=variable_shape if variable_shape else None)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 258, in call
return cls._variable_v1_call(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 219, in _variable_v1_call
shape=shape)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 197, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variable_scope.py", line 2596, in default_variable_creator
shape=shape)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 262, in call
return super(VariableMetaclass, cls).call(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py", line 1411, in init
distribute_strategy=distribute_strategy)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py", line 1542, in _init_from_args
initial_value() if init_from_fn else initial_value,
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py", line 122, in
init_val = lambda: initializer(shape, dtype=dtype)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py", line 425, in call
return self.random_generator.random_uniform(shape, -limit, limit, dtype)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py", line 788, in random_uniform
shape=shape, minval=minval, maxval=maxval, dtype=dtype, seed=self.seed)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/random_ops.py", line 273, in random_uniform
result = math_ops.add(rnd * (maxval - minval), minval, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_math_ops.py", line 343, in add
ops.raise_from_not_ok_status(e, name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 6606, in raise_from_not_ok_status
six.raise_from(core.status_to_exception(e.code, message), None)
File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add] name: conv2d_49/kernel/Initializer/random_uniform/
2020-09-15 16:28:22.672126: I tensorflow/core/common_runtime/bfc_allocator.cc:962] Sum Total of in-use chunks: 173.18MiB
2020-09-15 16:28:22.672152: I tensorflow/core/common_runtime/bfc_allocator.cc:964] total_region_allocated_bytes: 192258048 memory_limit: 192258048 available bytes: 0 curr_region_allocation_bytes: 268435456
2020-09-15 16:28:22.672265: I tensorflow/core/common_runtime/bfc_allocator.cc:970] Stats:
Limit: 192258048
InUse: 181590528
MaxInUse: 181590528
NumAllocs: 1232
MaxAllocSize: 34078720

2020-09-15 16:28:22.672343: W tensorflow/core/common_runtime/bfc_allocator.cc:429] ************_*******************xxxxxxxx
2020-09-15 16:28:22.690033: W tensorflow/core/framework/op_kernel.cc:1655] OP_REQUIRES failed at cwise_ops_common.cc:82 : Resource exhausted: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "object_tracker.py", line 163, in
app.run(main)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "object_tracker.py", line 52, in main
yolo = YoloV3(classes=FLAGS.num_classes)
File "/home/christopher/aiguyyolotest2/yolov3_deepsort/yolov3_tf2/models.py", line 211, in YoloV3
x_36, x_61, x = Darknet(name='yolo_darknet')(x)
File "/home/christopher/aiguyyolotest2/yolov3_deepsort/yolov3_tf2/models.py", line 78, in Darknet
x = DarknetBlock(x, 1024, 4)
File "/home/christopher/aiguyyolotest2/yolov3_deepsort/yolov3_tf2/models.py", line 67, in DarknetBlock
x = DarknetResidual(x, filters)
File "/home/christopher/aiguyyolotest2/yolov3_deepsort/yolov3_tf2/models.py", line 59, in DarknetResidual
x = DarknetConv(x, filters, 3)
File "/home/christopher/aiguyyolotest2/yolov3_deepsort/yolov3_tf2/models.py", line 49, in DarknetConv
use_bias=not batch_norm, kernel_regularizer=l2(0.0005))(x)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 748, in call
self._maybe_build(inputs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 2116, in _maybe_build
self.build(input_shapes)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/layers/convolutional.py", line 161, in build
dtype=self.dtype)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 446, in add_weight
caching_device=caching_device)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/tracking/base.py", line 744, in _add_variable_with_custom_getter
**kwargs_for_getter)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py", line 142, in make_variable
shape=variable_shape if variable_shape else None)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 258, in call
return cls._variable_v1_call(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 219, in _variable_v1_call
shape=shape)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 197, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variable_scope.py", line 2596, in default_variable_creator
shape=shape)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/variables.py", line 262, in call
return super(VariableMetaclass, cls).call(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py", line 1411, in init
distribute_strategy=distribute_strategy)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py", line 1542, in _init_from_args
initial_value() if init_from_fn else initial_value,
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer_utils.py", line 122, in
init_val = lambda: initializer(shape, dtype=dtype)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py", line 425, in call
return self._random_generator.random_uniform(shape, -limit, limit, dtype)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/init_ops_v2.py", line 788, in random_uniform
shape=shape, minval=minval, maxval=maxval, dtype=dtype, seed=self.seed)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/random_ops.py", line 273, in random_uniform
result = math_ops.add(rnd * (maxval - minval), minval, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_math_ops.py", line 343, in add
_ops.raise_from_not_ok_status(e, name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 6606, in raise_from_not_ok_status
six.raise_from(core._status_to_exception(e.code, message), None)
File "", line 3, in raise_from
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[3,3,512,1024] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [Op:Add] name: conv2d_49/kernel/Initializer/random_uniform/

Cannot connect to X server while running object_tracker.py

2021-05-29 07:06:31.158550: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2021-05-29 07:06:31.158681: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2021-05-29 07:06:31.158706: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2021-05-29 07:06:32.221407: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-05-29 07:06:32.253248: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-29 07:06:32.253858: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.75GiB deviceMemoryBandwidth: 298.08GiB/s
2021-05-29 07:06:32.254134: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2021-05-29 07:06:32.255794: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2021-05-29 07:06:32.258774: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2021-05-29 07:06:32.259119: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2021-05-29 07:06:32.260892: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2021-05-29 07:06:32.268283: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2021-05-29 07:06:32.272211: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-05-29 07:06:32.272334: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-29 07:06:32.272942: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-29 07:06:32.273447: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2021-05-29 07:06:32.274988: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
2021-05-29 07:06:32.279635: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2000189999 Hz
2021-05-29 07:06:32.279828: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x563d2f032840 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-05-29 07:06:32.279856: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-05-29 07:06:32.376068: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-29 07:06:32.376845: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x563d2f032a00 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-05-29 07:06:32.376878: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Tesla T4, Compute Capability 7.5
2021-05-29 07:06:32.377071: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-29 07:06:32.377606: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.75GiB deviceMemoryBandwidth: 298.08GiB/s
2021-05-29 07:06:32.377682: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2021-05-29 07:06:32.377716: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2021-05-29 07:06:32.377740: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2021-05-29 07:06:32.377764: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2021-05-29 07:06:32.377786: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2021-05-29 07:06:32.377806: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2021-05-29 07:06:32.377827: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-05-29 07:06:32.377901: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-29 07:06:32.378463: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-29 07:06:32.378975: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2021-05-29 07:06:32.379055: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2021-05-29 07:06:32.380231: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-05-29 07:06:32.380259: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0
2021-05-29 07:06:32.380270: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N
2021-05-29 07:06:32.380388: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-29 07:06:32.380990: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-29 07:06:32.381524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14257 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
2021-05-29 07:06:32.469772: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-29 07:06:32.470637: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:00:04.0 name: Tesla T4 computeCapability: 7.5
coreClock: 1.59GHz coreCount: 40 deviceMemorySize: 14.75GiB deviceMemoryBandwidth: 298.08GiB/s
2021-05-29 07:06:32.470749: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2021-05-29 07:06:32.470782: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2021-05-29 07:06:32.470807: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2021-05-29 07:06:32.470829: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2021-05-29 07:06:32.470850: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2021-05-29 07:06:32.470871: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2021-05-29 07:06:32.470894: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-05-29 07:06:32.471104: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-29 07:06:32.471960: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-29 07:06:32.472476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2021-05-29 07:06:32.472519: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-05-29 07:06:32.472533: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] 0
2021-05-29 07:06:32.472543: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0: N
2021-05-29 07:06:32.472657: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-29 07:06:32.473235: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:981] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-05-29 07:06:32.473748: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 14257 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
I0529 07:06:38.096785 139947425822592 object_tracker.py:55] weights loaded
I0529 07:06:38.098225 139947425822592 object_tracker.py:58] classes loaded
2021-05-29 07:06:39.522086: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2021-05-29 07:06:40.669860: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
: cannot connect to X server