joshi-bharat/deep_underwater_localization

Source Code for "DeepURL: Deep Pose Estimation Framework for Underwater Relative Localization", IROS 2020

Hi there, I am Bharat 👋

I am working on problems at the intersection of Robotics x Computer Vision at Autonomous Field Robotics Lab. My research is focused on increasing the robustness of visual-inertial SLAM for underwater environments. I have worked on projects involving comparing various state estimation algorithms in the underwater domain, developing a deep pose estimation framework for underwater relative localization, and using GoPro as an inexpensive sensor for underwater structure mapping. I recently finished working on robust underwater state estimation by swithing between less-accurate odometry estimate and accurate VIO to get track AUVs when VIO fails.

🔭 I’m currently working on tightly-coupled fusion of magnetometer with Visual-Inertial SLAM.

deep_underwater_localization's Issues

how to get absolutely accurate gt pose of the real robot?

hi, @joshi-bharat. I recently collected the gt pose of the real AUV. I use the 2D fiducial marker(ArUco tag) mounted on the AUV to calculate the relative pose, and do the reprojection of the 8 corners by R|T vetors. Unfortunately, the drawn 3d bounding box seems not right of some frames. So maybe the gt pose we collected by ArUco is not accurate. So how to get accurate gt pose of the real robot?

In addition，I checked the camera calibration, checked the ArUco detection codes, I cann't find any problem. it should be the limited accuracy of ArUco tag pose estimation in the underwater scene? I am not sure. Can you give me any advice to help me get the accuracy pose.? Sincerely thanks.

About simulation camera intrinsic matrix

Hello, first congratulations you on your new accepting paper. Then I have a question, wish to get your help.

Did you set the camera physical attribute the same as the gopro camera？ I found that you used the same camera intrinsic in the test_image_list.py source code. Theoretically，when calculate Rt_gt by pnp(https://github.com/joshi-bharat/deep_underwater_localization/blob/master/test_image_list.py#L189-L190), it should be the simulation camera intrinsic, right?

I run the train.py and then I got these errors. The environments are the same as README.md.

/home/juling/anaconda3/envs/py37/bin/python3.7 /home/juling/PycharmProjects/deepURL/train.py
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:

• https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
• https://github.com/tensorflow/addons
• https://github.com/tensorflow/io (for I/O related ops)
  If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/train.py:23: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py:1475: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/train.py:37: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
options available in V2.
- tf.py_function takes a python function which manipulates tf eager
tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
an ndarray (just call tensor.numpy()) but having access to eager tensors
means tf.py_functions can use accelerators such as GPUs as well as
being differentiable using a gradient tape.
- tf.numpy_function maintains the semantics of the deprecated tf.py_func
(it is not differentiable, and manipulates numpy arrays). It drops the
stateful argument making all functions stateful.

WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/train.py:44: The name tf.data.Iterator is deprecated. Please use tf.compat.v1.data.Iterator instead.

WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/train.py:44: DatasetV1.output_types (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_types(dataset).
WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/train.py:44: DatasetV1.output_shapes (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_shapes(dataset).
WARNING:tensorflow:From /home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/data/ops/iterator_ops.py:347: Iterator.output_types (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_types(iterator).
WARNING:tensorflow:From /home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/data/ops/iterator_ops.py:348: Iterator.output_shapes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_shapes(iterator).
WARNING:tensorflow:From /home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/data/ops/iterator_ops.py:350: Iterator.output_classes (from tensorflow.python.data.ops.iterator_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.data.get_output_classes(iterator).
WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/train.py:63: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:From /home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/contrib/layers/python/layers/layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/utils/layer_utils.py:86: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.

WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/model.py:294: The name tf.log is deprecated. Please use tf.math.log instead.

WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/train.py:72: The name tf.losses.get_regularization_loss is deprecated. Please use tf.compat.v1.losses.get_regularization_loss instead.

WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/train.py:75: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/train.py:78: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.

WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/train.py:91: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/utils/misc_utils.py:146: The name tf.train.piecewise_constant is deprecated. Please use tf.compat.v1.train.piecewise_constant instead.

WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/utils/misc_utils.py:154: The name tf.train.MomentumOptimizer is deprecated. Please use tf.compat.v1.train.MomentumOptimizer instead.

WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/train.py:107: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

Saving optimizer parameters to checkpoint! Remember to restore the global_step in the fine-tuning afterwards.
WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/train.py:120: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/train.py:123: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2021-07-22 15:39:40.089153: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-07-22 15:39:40.110998: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2699905000 Hz
2021-07-22 15:39:40.111317: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55bbbe9d3ae0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-07-22 15:39:40.111347: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2021-07-22 15:39:40.113514: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2021-07-22 15:39:40.226792: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-22 15:39:40.227612: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55bbbda483c0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-07-22 15:39:40.227627: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce 940MX, Compute Capability 5.0
2021-07-22 15:39:40.227814: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2021-07-22 15:39:40.228522: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce 940MX major: 5 minor: 0 memoryClockRate(GHz): 1.189
pciBusID: 0000:01:00.0
2021-07-22 15:39:40.228620: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory
2021-07-22 15:39:40.228674: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory
2021-07-22 15:39:40.228735: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory
2021-07-22 15:39:40.228799: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory
2021-07-22 15:39:40.228849: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory
2021-07-22 15:39:40.228898: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory
2021-07-22 15:39:40.228949: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory
2021-07-22 15:39:40.228956: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1641] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2021-07-22 15:39:40.228967: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-07-22 15:39:40.228973: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2021-07-22 15:39:40.228978: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/train.py:124: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.

WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/train.py:124: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.

INFO:tensorflow:Restoring parameters from /home/juling/PycharmProjects/deepURL/data/darknet_weights/yolov3.ckpt
WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/train.py:126: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.

WARNING:tensorflow:From /home/juling/PycharmProjects/deepURL/train.py:127: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.

----------- start to train -----------

0%| | 0/8228 [00:00<?, ?it/s]2021-07-22 15:40:42.426605: W tensorflow/core/framework/op_kernel.cc:1639] Unknown: AttributeError: 'NoneType' object has no attribute 'astype'
Traceback (most recent call last):

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in call
ret = func(*args)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 266, in get_batch_data
img_idx, img, y_true_13, y_true_26, y_true_52, singleshot, y_true_13_mask,y_true_26_mask, y_true_52_mask = parse_data(line, class_num, img_size, anchors, mode, letterbox_resize, nV=nV)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 195, in parse_data
img = random_color_distort(img)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 253, in random_color_distort
img = random_brightness(img, brightness_delta)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 247, in random_brightness
img = img.astype(np.float32)

AttributeError: 'NoneType' object has no attribute 'astype'

2021-07-22 15:40:42.426959: W tensorflow/core/framework/op_kernel.cc:1639] Unknown: AttributeError: 'NoneType' object has no attribute 'astype'
Traceback (most recent call last):

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in call
ret = func(*args)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 266, in get_batch_data
img_idx, img, y_true_13, y_true_26, y_true_52, singleshot, y_true_13_mask,y_true_26_mask, y_true_52_mask = parse_data(line, class_num, img_size, anchors, mode, letterbox_resize, nV=nV)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 195, in parse_data
img = random_color_distort(img)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 253, in random_color_distort
img = random_brightness(img, brightness_delta)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 247, in random_brightness
img = img.astype(np.float32)

AttributeError: 'NoneType' object has no attribute 'astype'

2021-07-22 15:40:42.428550: W tensorflow/core/framework/op_kernel.cc:1639] Unknown: AttributeError: 'NoneType' object has no attribute 'astype'
Traceback (most recent call last):

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in call
ret = func(*args)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 266, in get_batch_data
img_idx, img, y_true_13, y_true_26, y_true_52, singleshot, y_true_13_mask,y_true_26_mask, y_true_52_mask = parse_data(line, class_num, img_size, anchors, mode, letterbox_resize, nV=nV)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 195, in parse_data
img = random_color_distort(img)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 253, in random_color_distort
img = random_brightness(img, brightness_delta)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 247, in random_brightness
img = img.astype(np.float32)

AttributeError: 'NoneType' object has no attribute 'astype'

2021-07-22 15:40:43.336065: W tensorflow/core/framework/op_kernel.cc:1639] Unknown: AttributeError: 'NoneType' object has no attribute 'astype'
Traceback (most recent call last):

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in call
ret = func(*args)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 266, in get_batch_data
img_idx, img, y_true_13, y_true_26, y_true_52, singleshot, y_true_13_mask,y_true_26_mask, y_true_52_mask = parse_data(line, class_num, img_size, anchors, mode, letterbox_resize, nV=nV)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 195, in parse_data
img = random_color_distort(img)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 253, in random_color_distort
img = random_brightness(img, brightness_delta)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 247, in random_brightness
img = img.astype(np.float32)

AttributeError: 'NoneType' object has no attribute 'astype'

0%| | 0/8228 [00:38<?, ?it/s]
2021-07-22 15:40:43.340091: W tensorflow/core/framework/op_kernel.cc:1639] Unknown: AttributeError: 'NoneType' object has no attribute 'astype'
Traceback (most recent call last):

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in call
ret = func(*args)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 266, in get_batch_data
img_idx, img, y_true_13, y_true_26, y_true_52, singleshot, y_true_13_mask,y_true_26_mask, y_true_52_mask = parse_data(line, class_num, img_size, anchors, mode, letterbox_resize, nV=nV)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 195, in parse_data
img = random_color_distort(img)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 253, in random_color_distort
img = random_brightness(img, brightness_delta)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 247, in random_brightness
img = img.astype(np.float32)

AttributeError: 'NoneType' object has no attribute 'astype'

2021-07-22 15:40:43.340477: W tensorflow/core/framework/op_kernel.cc:1639] Unknown: AttributeError: 'NoneType' object has no attribute 'astype'
Traceback (most recent call last):

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in call
ret = func(*args)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 266, in get_batch_data
img_idx, img, y_true_13, y_true_26, y_true_52, singleshot, y_true_13_mask,y_true_26_mask, y_true_52_mask = parse_data(line, class_num, img_size, anchors, mode, letterbox_resize, nV=nV)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 195, in parse_data
img = random_color_distort(img)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 253, in random_color_distort
img = random_brightness(img, brightness_delta)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 247, in random_brightness
img = img.astype(np.float32)

AttributeError: 'NoneType' object has no attribute 'astype'

2021-07-22 15:40:43.537734: W tensorflow/core/framework/op_kernel.cc:1639] Unknown: AttributeError: 'NoneType' object has no attribute 'astype'
Traceback (most recent call last):

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in call
ret = func(*args)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 266, in get_batch_data
img_idx, img, y_true_13, y_true_26, y_true_52, singleshot, y_true_13_mask,y_true_26_mask, y_true_52_mask = parse_data(line, class_num, img_size, anchors, mode, letterbox_resize, nV=nV)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 195, in parse_data
img = random_color_distort(img)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 253, in random_color_distort
img = random_brightness(img, brightness_delta)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 247, in random_brightness
img = img.astype(np.float32)

AttributeError: 'NoneType' object has no attribute 'astype'

2021-07-22 15:40:43.742177: W tensorflow/core/framework/op_kernel.cc:1639] Invalid argument: TypeError: '>=' not supported between instances of 'NoneType' and 'int'
Traceback (most recent call last):

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in call
ret = func(*args)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 266, in get_batch_data
img_idx, img, y_true_13, y_true_26, y_true_52, singleshot, y_true_13_mask,y_true_26_mask, y_true_52_mask = parse_data(line, class_num, img_size, anchors, mode, letterbox_resize, nV=nV)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 195, in parse_data
img = random_color_distort(img)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 253, in random_color_distort
img = random_brightness(img, brightness_delta)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 250, in random_brightness
return np.clip(img, 0, 255)

File "<array_function internals>", line 6, in clip

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 2084, in clip
return _wrapfunc(a, 'clip', a_min, a_max, out=out, **kwargs)

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 58, in _wrapfunc
return _wrapit(obj, method, *args, **kwds)

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 47, in _wrapit
result = getattr(asarray(obj), method)(*args, **kwds)

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/numpy/core/_methods.py", line 132, in _clip
um.clip, a, min, max, out=out, casting=casting, **kwargs)

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/numpy/core/_methods.py", line 85, in _clip_dep_invoke_with_casting
return ufunc(*args, out=out, **kwargs)

TypeError: '>=' not supported between instances of 'NoneType' and 'int'

2021-07-22 15:40:43.742373: W tensorflow/core/framework/op_kernel.cc:1639] Invalid argument: TypeError: '>=' not supported between instances of 'NoneType' and 'int'
Traceback (most recent call last):

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in call
ret = func(*args)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 266, in get_batch_data
img_idx, img, y_true_13, y_true_26, y_true_52, singleshot, y_true_13_mask,y_true_26_mask, y_true_52_mask = parse_data(line, class_num, img_size, anchors, mode, letterbox_resize, nV=nV)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 195, in parse_data
img = random_color_distort(img)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 253, in random_color_distort
img = random_brightness(img, brightness_delta)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 250, in random_brightness
return np.clip(img, 0, 255)

File "<array_function internals>", line 6, in clip

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 2084, in clip
return _wrapfunc(a, 'clip', a_min, a_max, out=out, **kwargs)

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 58, in _wrapfunc
return _wrapit(obj, method, *args, **kwds)

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 47, in _wrapit
result = getattr(asarray(obj), method)(*args, **kwds)

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/numpy/core/_methods.py", line 132, in _clip
um.clip, a, min, max, out=out, casting=casting, **kwargs)

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/numpy/core/_methods.py", line 85, in _clip_dep_invoke_with_casting
return ufunc(*args, out=out, **kwargs)

TypeError: '>=' not supported between instances of 'NoneType' and 'int'

2021-07-22 15:40:43.937299: W tensorflow/core/framework/op_kernel.cc:1639] Unknown: AttributeError: 'NoneType' object has no attribute 'astype'
Traceback (most recent call last):

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in call
ret = func(*args)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 266, in get_batch_data
img_idx, img, y_true_13, y_true_26, y_true_52, singleshot, y_true_13_mask,y_true_26_mask, y_true_52_mask = parse_data(line, class_num, img_size, anchors, mode, letterbox_resize, nV=nV)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 195, in parse_data
img = random_color_distort(img)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 253, in random_color_distort
img = random_brightness(img, brightness_delta)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 247, in random_brightness
img = img.astype(np.float32)

AttributeError: 'NoneType' object has no attribute 'astype'

Traceback (most recent call last):
File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: {{function_node _inference_Dataset_map_146}} AttributeError: 'NoneType' object has no attribute 'astype'
Traceback (most recent call last):

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in call
ret = func(*args)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 266, in get_batch_data
img_idx, img, y_true_13, y_true_26, y_true_52, singleshot, y_true_13_mask,y_true_26_mask, y_true_52_mask = parse_data(line, class_num, img_size, anchors, mode, letterbox_resize, nV=nV)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 195, in parse_data
img = random_color_distort(img)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 253, in random_color_distort
img = random_brightness(img, brightness_delta)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 247, in random_brightness
img = img.astype(np.float32)

AttributeError: 'NoneType' object has no attribute 'astype'

 [[{{node PyFunc}}]]
 [[IteratorGetNext]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/juling/PycharmProjects/deepURL/train.py", line 143, in
feed_dict={is_training: True})
File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: {{function_node _inference_Dataset_map_146}} AttributeError: 'NoneType' object has no attribute 'astype'
Traceback (most recent call last):

File "/home/juling/anaconda3/envs/py37/lib/python3.7/site-packages/tensorflow_core/python/ops/script_ops.py", line 235, in call
ret = func(*args)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 266, in get_batch_data
img_idx, img, y_true_13, y_true_26, y_true_52, singleshot, y_true_13_mask,y_true_26_mask, y_true_52_mask = parse_data(line, class_num, img_size, anchors, mode, letterbox_resize, nV=nV)

File "/home/juling/PycharmProjects/deepURL/utils/data_utils.py", line 195, in parse_data
img = random_color_distort(img)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 253, in random_color_distort
img = random_brightness(img, brightness_delta)

File "/home/juling/PycharmProjects/deepURL/utils/data_aug.py", line 247, in random_brightness
img = img.astype(np.float32)

AttributeError: 'NoneType' object has no attribute 'astype'

 [[{{node PyFunc}}]]
 [[IteratorGetNext]]

Process finished with exit code 1

About how to make labels for my own auv model

@joshi-bharat Sorry, I'm here again. Now I'm going to apply the deepURL to my AUV. I have some questions when making labels.

The aqua_glass_removed.ply file is only used during testing, isn't it? I don't seem to see the use of this file in train.py. So what does it do？ I have my own SolidWorks model of AUV, so do I just need to export to ply format and replace your ply file in the code? I'm a little confused.

Another problem, If I get the pose from the simulation and get the 3D coordinates of 8 key points, do I still need to use the camera's internal parameter matrix to obtain the 2D projection? Or do I get the 2D projection of 8 key points directly from the simulation? I'm a little confused.

One more question, the checkpoint of Yolo V3 is also used in train.py. If it is applied to my own AUV, do I need to find a tensorflow code of Yolo V3 to realize object detection and train to get the weight to identify my own AUV?

I am looking forward to your reply. Thanks!

no refraction？

Hi,
does refraction have some sorts of effects on the pose estimation?

About dataset label

Hello.

I noticed that only the 2D projections are provided in the dataset, which is truncated into integers. I was wondering if the ground-truth pose or the original 2D projections can be provided.

Thank you!

Maybe I found a bug in the source code

Hi, @joshi-bharat. it looks like that I found a bug in the test code.

For test_images_list.py, when you defined compute graph, you called pose_loss.predict() function (see code) to get filted x,y,conf, selected values. It looks fine. However, you used if-else structure (see code) to control data flow in the pose_loss.predict() function. In fact, compute graph is static in tensorflow1.0 . So the program always executes only the else branch (see code), regardless of whether bbox_mask is None. So bbox_masks don't work in fact.

By the way, the conf here does not use the sigmoid fuction. So the conf isn't in 0~1, that results validmask to always be True.

About the bbox_mask

I found another problem that the code can't get right bbox_masks to filt x_, y_, conf_. You used predicted box to get bbox_masks in the computer graph. Fist 'if bbox is None' can't be used, for the graph is statics. Second, input value bbox is tensor type，can't use 'for box in bbox' or ' // 'or 'int( )' and other gramma. And How to solve this problem? We need to use Tensor type to get bbox_masks from bbox in the computer graph.

Cannot convert a symbolic Tensor (meshgrid/Size_1:0) to a numpy array.

The python version is 3.7.5 and tensorflow version is 1.15.0. While running test_single_imapy.py file i got this error. I am unable to solve that error. waiting for your kind reponse.

Traceback (most recent call last):
File "test_single_image.py", line 82, in
pred_boxes, pred_confs, pred_probs = yolo_model.predict(yolo_features)
File "/home/waqar/deepurl3.7/deep_underwater_localization-master/model.py", line 179, in predict
reorg_results = [self.reorg_layer(feature_map, anchors) for (feature_map, anchors) in feature_map_anchors]
File "/home/waqar/deepurl3.7/deep_underwater_localization-master/model.py", line 179, in
reorg_results = [self.reorg_layer(feature_map, anchors) for (feature_map, anchors) in feature_map_anchors]
File "/home/waqar/deepurl3.7/deep_underwater_localization-master/model.py", line 139, in reorg_layer
grid_x, grid_y = tf.meshgrid(grid_x, grid_y)
File "/home/waqar/deepurl3.7/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py", line 2943, in meshgrid
mult_fact = ones(shapes, output_dtype)
File "/home/waqar/deepurl3.7/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py", line 2560, in ones
output = _constant_if_small(one, shape, dtype, name)
File "/home/waqar/deepurl3.7/lib/python3.7/site-packages/tensorflow_core/python/ops/array_ops.py", line 2295, in _constant_if_small
if np.prod(shape) < 1000:
File "<array_function internals>", line 6, in prod
File "/home/waqar/deepurl3.7/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 3052, in prod
keepdims=keepdims, initial=initial, where=where)
File "/home/waqar/deepurl3.7/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 86, in _wrapreduction
return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
File "/home/waqar/deepurl3.7/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 736, in array
" array.".format(self.name))
NotImplementedError: Cannot convert a symbolic Tensor (meshgrid/Size_1:0) to a numpy array.

CycleGAN

Hello, I want to know whether the CycleGAN mentioned in the article is open source?

About cycle GAN to synthesize pictures

Hello, thank you for your help before. I'm here to disturb you again.

I have completed the acquisition of the rendered image in the simulation, and successfully obtained the 2D coordinates of the 8 corner points of the 3D model, and now I am making synthetic images. Regarding cycle GAN, I would like to ask you a few questions.

• Do you directly use the code of junyanzhu to complete the synthesis of the rendered image? Have you made any changes to the source code？Or just use your own datasets, completely using the source code. Refer to https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix.
• It seems that the resolution of the pictures synthesized by cycle GAN is very low. How did you get the final high-quality pictures? It looks like your synthetic pictures looks good.

Thanks to you again!