opendrivelab/thinktwice

[CVPR 2023] Pytorch implementation of ThinkTwice, a SOTA Decoder for End-to-end Autonomous Driving under BEV.

thinktwice's Issues

ImportError: cannot import name 'voxel_pooling_ext'

How to solve this erorr?

Question about the bounding box 3d visulization.

Hi, thanks for the great work! I want to visualize the 3d bounding box. I use the bounding box information in the 3d_bbs json files. Are bounding boxes in the lidar coord? I use lidar2img to project the boxes and got results like this.
What should I do to get the correct boxes?

About collect images for calibration.

Hello,
I followed the instructions in https://github.com/OpenDriveLab/ThinkTwice/blob/main/camera_calibration/README.md and spent several days installing UE4 and Carla. However, when I run collect_data_for_calibration.py, I found that there is no calibration checkerboard in the scene, and the camera perspective does not change. Each frame collected is the same, which cannot be used to calculate distortion parameters with calculate_distortion_parameters.py.
Could you please provide any suggestions about this situation?

Additionally, I originally planned to change the camera resolution to 900x256, FOV to 100, front_camera position to x=1.3,y=0,z=2.5 and set undistort to False, so I don't need to calculate distortion parameters. But there are still some parameters in the transformation matrices that I don't know how to set.

Regarding LIDAR2IMG:

LIDAR2IMG = {
    "rgb_front":np.array([[800.0, 214.35935394, 0.0, -1200.0], [450.0, 0.0, -214.35935394, -139.10161515000004], [1.0, 0.0, 0.0, -1.5], [0.0, 0.0, 0.0, 1.0]]),

I can see that 800 and 450 are half of the original width 1600 and height 900, and 214.359 corresponds to parameters that can be directly calculated from the image resolution and FOV. After changing the camera settings, this should be changed to 377.594.

The value of 1200 seems to be related to the camera position, so I still can make a guess about what value should be set for this parameter after changing the camera settings, But I don't know what 139.1016 represents. Therefore, I plan to follow the instructions in https://github.com/OpenDriveLab/ThinkTwice/blob/main/camera_calibration/README.md to see if it can be calculated by calculate_distortion_parameters.py. However, it seems that there are some problems with the process of collecting images for calibration, which is a bit confusing.

Before possible reply, I will continue to try to solve this problem by myself. If I can solve it successfully, I will close this issue on my own.

Error when collecting data

Running the route
initialized
XIO: fatal IO error 11 (Resource temporarily unavailable) on X server "localhost:10.0"
after 4060 requests (4060 known processed) with 0 events remaining.
terminating with uncaught exception of type std::__1::system_error: mutex lock failed: Invalid argument
Signal 6 caught.
/data/carla/CarlaUE4.sh: line 5: 555 Segmentation fault "$UE4_PROJECT_ROOT/CarlaUE4/Binaries/Linux/CarlaUE4-Linux-Shipping" CarlaUE4 $@
Traceback (most recent call last):
File "/data/ThinkTwice/leaderboard/leaderboard/envs/sensor_interface.py", line 244, in get_data
sensor_data = self._new_data_buffers.get(True, self._queue_timeout)
File "/home/wangshihao02/.conda/envs/thinktwice/lib/python3.7/queue.py", line 178, in get
raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/data/ThinkTwice/leaderboard/leaderboard/scenarios/scenario_manager.py", line 154, in _tick_scenario
ego_action = self._agent()
File "/data/ThinkTwice/leaderboard/leaderboard/autoagents/agent_wrapper.py", line 82, in call
return self._agent()
File "/data/ThinkTwice/leaderboard/leaderboard/autoagents/autonomous_agent.py", line 104, in call
input_data = self.sensor_interface.get_data()
File "/data/ThinkTwice/leaderboard/leaderboard/envs/sensor_interface.py", line 248, in get_data
raise SensorReceivedNoData("A sensor took too long to send their data")
leaderboard.envs.sensor_interface.SensorReceivedNoData: A sensor took too long to send their data

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "leaderboard/leaderboard/leaderboard_evaluator.py", line 542, in main
leaderboard_evaluator.run(arguments)
File "leaderboard/leaderboard/leaderboard_evaluator.py", line 462, in run
self._load_and_run_scenario(args, config)
File "leaderboard/leaderboard/leaderboard_evaluator.py", line 416, in _load_and_run_scenario
self.manager.run_scenario()
File "/data/ThinkTwice/leaderboard/leaderboard/scenarios/scenario_manager.py", line 138, in run_scenario
self._tick_scenario(timestamp)
File "/data/ThinkTwice/leaderboard/leaderboard/scenarios/scenario_manager.py", line 158, in _tick_scenario
raise RuntimeError(e)
RuntimeError: A sensor took too long to send their data
Collect Done
Hi, thanks for the great work!
I get a runtime error when collecting the data.
Would you please provide some advise to avoid this?

speeds of other vehicles

Hello, I noticed that when collecting data, the speeds of other vehicles obtained were negative. Do you know what this is? Speed obtained through function _get_forward_speed()

questions about LIDAR2CAM

Thanks for this great work. In the file open_loop_training/code/datasets/pipelines/transform.py, the LIDAR2CAM is used. The problem is when i use the sensor parameter in leaderboard/team_code/roach_ap_agent_data_collection.py to compute LIDAR2CAM transform matrix, these two results are different.

For example, the rgb_front in LIDAR2CAM is
"rgb_front": np.array([ [0.0, 1.0, 0.0, 0.0] ,
[0.0, 0.0, -1.0, 2.5] ,
[1.0, 0.0, 0.0, -1.5] ,
[0.0, 0.0, 0.0, 1.0] ]),

The result using the transform T_ego2cam* T_lidar2ego (where T_ego2cam = inverse(T_cam2ego) ) is
rgb_front [[ 1. 0. 0. -1.5]
[ 0. 1. 0. 0. ]
[ 0. 0. 1. 0. ]
[ 0. 0. 0. 1. ]]

. Besides the rgb_front paramters, the LIDAR2CAM parameters of rgb_left, rgb_right, rgb_back are also different. Is there something missing when I compute T_lidar2cam?

the code is in the following

import os
import numpy as np

from nuscenes.eval.common.utils import quaternion_yaw, Quaternion

sensor_data = [
                {
                    'type': 'sensor.camera.rgb',
                    'x': 1.5, 'y': 0.0, 'z':2.5,
                    'roll': 0.0, 'pitch': 0.0, 'yaw': 0.0,
                    'width': 1600, 'height': 900, 'fov': 150,
                    'id': 'rgb_front'
                    },
                {
                    'type': 'sensor.camera.rgb',
                    'x': 0, 'y': -0.3, 'z': 2.5,
                    'roll': 0.0, 'pitch': 0.0, 'yaw': -90.0,
                    'width': 1600, 'height': 900, 'fov': 150,
                    'id': 'rgb_left'
                    },
                {
                    'type': 'sensor.camera.rgb',
                    'x': 0, 'y': 0.3, 'z': 2.5,
                    'roll': 0.0, 'pitch': 0.0, 'yaw': 90.0,
                    'width': 1600, 'height': 900, 'fov': 150,
                    'id': 'rgb_right'
                    },
                {
                    'type': 'sensor.camera.rgb',
                    'x': -1.6, 'y': 0.0, 'z': 2.5,
                    'roll': 0.0, 'pitch': 0.0, 'yaw': 180.0,
                    'width': 1600, 'height': 900, 'fov': 150,
                    'id': 'rgb_back'
                    },
                {   'type': 'sensor.lidar.ray_cast',
                'x': 0.0, 'y': 0.0, 'z': 2.5,
                'roll': 0.0, 'pitch': 0.0, 'yaw': 0.0,
                'id': 'lidar'
                },
]

def get_rotation_matrix(info):
    r_roll = Quaternion(axis =  [1, 0, 0], radians = np.deg2rad(si['roll'])).rotation_matrix
    r_pitch = Quaternion(axis =  [0, 1, 0], radians = np.deg2rad(si['pitch'])).rotation_matrix
    r_yaw = Quaternion(axis = [0, 0, 1], radians = np.deg2rad(si['yaw'])).rotation_matrix

    r = np.array(r_yaw) @ np.array(r_pitch) @ np.array(r_roll)

    return Quaternion(matrix = r), r


sensor_transform = {}
for si in sensor_data:

    res = {}
    res['id'] = si['id']
    res['translation'] = np.array([si['x'], si['y'], si['z'] ])
    res['rotation'],_ =  get_rotation_matrix(si)
    #sensor_transform.append(res)
    sensor_transform[si['id']] = res

name = 'lidar'
sub_sensor_info = sensor_transform[name]
lidar2ego = np.eye(4)
lidar2ego[:3,:3] = np.array(Quaternion(sub_sensor_info['rotation']).rotation_matrix)
lidar2ego[:3, 3] = sub_sensor_info['translation']


for name in ['rgb_front','rgb_left',  'rgb_right', 'rgb_back']:
    sub_sensor_info = sensor_transform[name]

    sensor2ego = np.eye(4)
    sensor2ego[:3,:3] = np.array(Quaternion(sub_sensor_info['rotation']).rotation_matrix)
    sensor2ego[:3, 3] = sub_sensor_info['translation']


    lidar2sensor = np.linalg.inv(sensor2ego) @ lidar2ego
    print( name,  lidar2sensor)

The results are :

rgb_front [[ 1.   0.   0.  -1.5]
 [ 0.   1.   0.   0. ]
 [ 0.   0.   1.   0. ]
 [ 0.   0.   0.   1. ]]
rgb_left [[ 2.22044605e-16 -1.00000000e+00  0.00000000e+00 -3.00000000e-01]
 [ 1.00000000e+00  2.22044605e-16  0.00000000e+00  6.66133815e-17]
 [ 0.00000000e+00  0.00000000e+00  1.00000000e+00  0.00000000e+00]
 [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  1.00000000e+00]]
rgb_right [[ 2.22044605e-16  1.00000000e+00  0.00000000e+00 -3.00000000e-01]
 [-1.00000000e+00  2.22044605e-16  0.00000000e+00 -6.66133815e-17]
 [ 0.00000000e+00  0.00000000e+00  1.00000000e+00  0.00000000e+00]
 [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  1.00000000e+00]]
rgb_back [[-1.00000000e+00  1.22464680e-16  0.00000000e+00 -1.60000000e+00]
 [-1.22464680e-16 -1.00000000e+00  0.00000000e+00 -1.95943488e-16]
 [ 0.00000000e+00  0.00000000e+00  1.00000000e+00  0.00000000e+00]
 [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  1.00000000e+00]]

the inference time

Hi, I'd like to know the inference time of your model. Could you provide the inference time on a single GPU.

_get_3d_bbs

Hi，When collecting data, does the function _get_3d_bbs(self, lidar=None, max_distance=50) obtain vehicles within the range of the ego camera or all vehicles in the world? If it is the former, I don't seem to have written any relevant filtering functions in the code?

Research details

Hi Team,

I am starting my research in the autonomous driving field, could you help me with the following questions about your architecture.

1. I would like to know what are the inputs that architecture expects and what exactly are the outputs generated from the architecture.
2. I see a video in the youtube of the trained model, it looks amazing. I am just curious to know if you are predicting both future waypoints and control commands. If you are predicting only the future waypoints how are you managing to stop vehicle in case of any obstacle.

Thanks in advance

Question about .register_module() function

Hello, I noticed in the mmlab tutorial that when registering a custom module, it seems that modifying the corresponding __init__.py file is required for the model to be successfully registered and used by the build_model() function in other files. However, I didn't see such operation in thinktwice. Could you please tell me how is it achieved?

@DETECTORS.register_module()
class EncoderDecoder(BaseModule):
    def __init__(self,  
                img_encoder,
                decoder,
                lidar_encoder=None,
                num_cams = 4,
                use_depth=False,
                use_seg=False,
                downsample_factor=16,
                seg_downsample_factor=2,
                train_cfg=None,
                test_cfg=None,
                ):
        super().__init__()

The versions of the nvidia gpu, driver, cuda, cudnn?

I am trying to start the carla 0.9.10 docker on the gpu 3090 (driver 460.27). However, the 9.10 docker fails to start because the driver is too new. Could you please tell these parameters when conducting the think-twice experiments?

'3d_bbs'

What is the purpose of saving data for '3d_bbs'?

Eval error: Segmentation fault (core dumped)

When I want to evaluation ThinkTwice on Town05 Long, some error appear:
`Fatal Python error: Segmentation fault

Thread 0x00007f3ba6260700 (most recent call first):
File "/home/shannon/miniconda3/envs/thinktwice/lib/python3.7/threading.py", line 300 in wait
File "/home/shannon/miniconda3/envs/thinktwice/lib/python3.7/threading.py", line 552 in wait
File "/home/shannon/miniconda3/envs/thinktwice/lib/python3.7/threading.py", line 1175 in run
File "/home/shannon/miniconda3/envs/thinktwice/lib/python3.7/threading.py", line 926 in _bootstrap_inner
File "/home/shannon/miniconda3/envs/thinktwice/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007f3ba5a5f700 (most recent call first):
File "/home/shannon/ThinkTwice/leaderboard/leaderboard/envs/sensor_interface.py", line 77 in run
File "/home/shannon/miniconda3/envs/thinktwice/lib/python3.7/threading.py", line 870 in run
File "/home/shannon/miniconda3/envs/thinktwice/lib/python3.7/threading.py", line 926 in _bootstrap_inner
File "/home/shannon/miniconda3/envs/thinktwice/lib/python3.7/threading.py", line 890 in _bootstrap

Current thread 0x00007f3d50558740 (most recent call first):
File "/home/shannon/miniconda3/envs/thinktwice/lib/python3.7/site-packages/shapely/geometry/base.py", line 194 in _empty
File "/home/shannon/miniconda3/envs/thinktwice/lib/python3.7/site-packages/shapely/geometry/base.py", line 209 in del
File "/home/shannon/ThinkTwice/scenario_runner/srunner/scenariomanager/scenarioatomics/atomic_criteria.py", line 1806 in update
File "/home/shannon/miniconda3/envs/thinktwice/lib/python3.7/site-packages/py_trees/behaviour.py", line 249 in tick
File "/home/shannon/miniconda3/envs/thinktwice/lib/python3.7/site-packages/py_trees/composites.py", line 578 in tick
File "/home/shannon/miniconda3/envs/thinktwice/lib/python3.7/site-packages/py_trees/composites.py", line 578 in tick
File "/home/shannon/miniconda3/envs/thinktwice/lib/python3.7/site-packages/py_trees/behaviour.py", line 158 in tick_once
File "/home/shannon/ThinkTwice/leaderboard/leaderboard/scenarios/scenario_manager.py", line 166 in _tick_scenario
File "/home/shannon/ThinkTwice/leaderboard/leaderboard/scenarios/scenario_manager.py", line 138 in run_scenario
File "leaderboard/leaderboard/leaderboard_evaluator.py", line 417 in _load_and_run_scenario
File "leaderboard/leaderboard/leaderboard_evaluator.py", line 463 in run
File "leaderboard/leaderboard/leaderboard_evaluator.py", line 543 in main
File "leaderboard/leaderboard/leaderboard_evaluator.py", line 552 in
Segmentation fault (core dumped)`

Do you meet this problem when eval the ckpt provided by GoogleDrive?

Carla failed to start on remote server

Run the generate data command on the remote server, and the process will exit directly after displaying the following log：

4.24.3-0+++UE4+Release-4.24 518 0
Disabling core dumps.

data generation

Hi，when I run the roach_agent to collecte data, but reported this error：

AttributeError: 'NoneType' object has no attribute 'get_location'
leaderboard.autoagents.agent_wrapper.AgentError: 'NoneType' object has no attribute 'get_location'

Why?

when I compile CUDA function for LSS from BEVDepth，it repears the problem like followings:

when I compile CUDA function for LSS from BEVDepth，it repears the problem like followings:
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda-11.3/'
Traceback (most recent call last):
File "setup.py", line 44, in
sources_cuda=['src/voxel_pooling_forward_cuda.cu'],
File "setup.py", line 27, in make_cuda_ext
raise EnvironmentError('CUDA is required to compile MMDetection!')
OSError: CUDA is required to compile MMDetection!
How can I solve it ?

Performance gap in reproducing results

The reproduced results show that DS is much lower than the numerical value in the paper. Is this result jitter normal?

Does ThinkTwice have Evaluation on Carla Leaderboard?

In paper, the method is evaluated on Town05 Long, however most of End-End method may report the eval result on Carla Leaderboard. So I want to know does ThinkTwice have evaluation on Carla Leaderboard?

KeyError: 'BENCHMARK'

Hello, I want to prepare the dataset and run CUDA_VISIBLE_DEVICES=0 nohup bash ./leaderboard/scripts/collect_data.sh 22023 22033 town05_00 False True roach_ap_agent_data_collection all_towns_traffic_scenarios_no256 2>&1 > town05_00.log &
However, the town05_00.log appear the following error:
Traceback (most recent call last):
File "leaderboard/leaderboard/leaderboard_evaluator.py", line 40, in
from leaderboard.utils.statistics_manager import StatisticsManager
File "/home/user/ThinkTwice-main/leaderboard/leaderboard/utils/statistics_manager.py", line 28, in
if os.environ["BENCHMARK"] == "longest6":
File "/home/user/anaconda3/envs/thinktwice/lib/python3.7/os.py", line 681, in getitem
raise KeyError(key) from None
KeyError: 'BENCHMARK'

How can I fix it? Thanks.

How to set parameters of transform.py?

Thanks for your amazing work!
If I want to use a single RGB camera and set the resolution to a smaller size, for example, changing it from the default 1600x900 to 900x256, do I need to make corresponding modifications to the parameters in the

open_loop_training/code/datasets/pipelines/transform.py

file? How should I modify it? Alternatively, if the settings in transform.py for 1600x900 would cause issues at the 900x256 resolution, can I skip this transformation process altogether?

More details on the Longest6 benchmark

Hi authors, thank you for your excellent work!

We would like to cite ThinkTwice as a comparison method, but we evaluated our approach on the Longest6 benchmark. We noticed that you only conducted a single evaluation in your paper. We attempted to run your ckpt model, which you provided in this repository, on the Longest6 benchmark. However, we encountered a challenge as your model demands significant computing resources, which our team cannot afford.

Therefore, we would like to inquire whether you have conducted multiple evaluations on the Longest6 benchmark or if you could offer us some advice on how to proceed.