yizhou-wang/rodnet

RODNet: Radar object detection network

Hi there 👋

• 🔭 I'm currently a Deep Learning Engineer at NVIDIA.
• 🌱 I’m interested in autonomous driving, sensor fusion, cross-modal object detection and tracking, and 3D computer vision.

rodnet's Issues

Confidence greater than 1(BUG)

RODNet predicts a class with conf > 1, is it common?

File "/media/chuan/MyPassport/deep_learning/cruw-devkit/cruw/eval/metrics.py", line 21, in get_ols_btw_objects
"obj1['score'] = %s, obj2['score'] = %s" % (obj1['score'], obj2['score']))
TypeError: Confidence score of obj1 should not be smaller than obj2. obj1['score'] = 1.0, obj2['score'] = 1.055537462234497

'annotation' folder is missing under cruw folder

Hi Yizhou,

I'm testing your prepare_data.py script to generate tranining data. However, it throws out some error at the following line:

RODNet/tools/prepare_dataset/prepare_data.py

Line 10 in 969cad6

from cruw.annotation.init_json import init_meta_json

I guess you have forgotten to put the annotation folder under cruw project.

cdc模型，测试验证集的时候，有的类别得分输出概率大于1

您好！
感谢您的源码。我在测试cdc模型的时候，有的类别得分输出大于1，导致评测AP和AR的时候会报错：Confidence score of obj1 should not be smaller than obj2. ""obj1['score'] = %s, obj2['score'] = %s"

后来我查看cdc模型和hgwi模型的decode结构时，发现二者有差异：
cdc模型decode最后没有加prelu激活：
x = self.prelu(self.convt1(x)) # (B, 256, W/4, 16, 16) -> (B, 128, W/2, 32, 32)
x = self.prelu(self.convt2(x)) # (B, 128, W/2, 32, 32) -> (B, 64, W, 64, 64)
x = self.convt3(x) # (B, 64, W, 64, 64) -> (B, 3, W, 128, 128)
return x

而hgwi模型最后有加prelu激活：
x = self.prelu(self.convt1(x + x3)) # (B, 256, W/4, 16, 16) -> (B, 128, W/2, 32, 32)
x = self.prelu(self.conv1(x))
x = self.prelu(self.convt2(x + x2)) # (B, 128, W/2, 32, 32) -> (B, 64, W, 64, 64)
x = self.prelu(self.conv2(x))
x = self.prelu(self.convt3(x + x1)) # (B, 64, W, 64, 64) -> (B, 3, W, 128, 128)
x = self.prelu(self.conv3(x))
return x

但是原论文的模型示意图上面，所有模型decode模块的最后都加了prelu激活，不知这是否是一个code bug？
感谢！

Radar RF images

Hi, this is a good job!

Can you provide pre-processing code that obtains final RF images?

Thanks!

关于测试结果

您好~我刚刚在CodaLab测试了我的结果，请问scores.txt中的PL、CR、CS和HW分别代表什么呢？

关于ex_evaluate_rod2021.py

请问代码中submit_dir指的是什么路径？

CRF部分的代码

您好？请问您可以提供一份CRF注释的代码吗？

How is the variance for each class calculated

Although discussed in #2, I am still confused about how variance is computed. In the paper, it says "the variance is related to the object class and scale information", while in the code, sigma is calculated by

RODNet/rodnet/core/confidence_map/generate.py

Line 36 in 969cad6

sigma = 2 * np.arctan(confmap_length[class_name] / (2 * range_grid[rng_idx])) * confmap_sigmas[class_name]

As I understand, sigma is the class-specific variance, and confmap_sigma is a hyper-parameter for variance per class. My question is:

• Is there a physical meaning or intuition behind np.arctan(confmap_length[class_name] / (2 * range_grid[rng_idx]))
• Is confmap_sigmas, confmap_sigmas_interval and confmap_length all hyper-parameters choosed via experiments?

Thanks in advance!

CUDA error: an illegal memory access was encountered

Traceback (most recent call last):
File "tools/train.py", line 235, in
confmap_preds = rodnet(data.float().cuda())
File "/share/home/3120305377/.conda/envs/rodenet2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/share/home/3120305377/PythonProjects/RODNet/rodnet/models/rodnet_hgwi_v2.py", line 35, in forward
out = self.stacked_hourglass(x)
File "/share/home/3120305377/.conda/envs/rodenet2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/share/home/3120305377/PythonProjects/RODNet/rodnet/models/backbones/hgwi.py", line 46, in forward
x, x1, x2, x3 = self.hourglass[i]0
File "/share/home/3120305377/.conda/envs/rodenet2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/share/home/3120305377/PythonProjects/RODNet/rodnet/models/backbones/hgwi.py", line 130, in forward
x1 = self.relu(self.skip_bn1(self.skip_inception1(x)))
File "/share/home/3120305377/.conda/envs/rodenet2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/share/home/3120305377/PythonProjects/RODNet/rodnet/models/backbones/hgwi.py", line 89, in forward
branch2 = self.branch2b(branch2)
File "/share/home/3120305377/.conda/envs/rodenet2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/share/home/3120305377/.conda/envs/rodenet2/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 479, in forward
return F.conv3d(input, self.weight, self.bias, self.stride,
RuntimeError: CUDA error: an illegal memory access was encountered

1、I have encountered this Error on A100 40GB. After google, Someone said that it might be the dataset and model were placed on the CPU and GPU, respectively. So, What exactly is the problem?
2、Or, would you like to provide trained models for testing?
Thanks a lot.

Using processed pointclouds from Radar data

I have a question about the ability to use the received radar point clouds from raw radar data as an input to your model.
Is it possible here? Any suggestions?

Thanks!

radar

radar

Compile pip install -e . with error

pip install -e reports an error. According to the online solution is not solved, help!The following screenshots of the error are attached, without the screenshots of Requirement already satisfied and Warning!

can't create the test/.pkl file

Hi, for my understanding, the "prepare.py" will generate training data(image and radar) and testing data(only radar). But the code seems to try to generate both even when I deal with the testing data. then it will raise the error(no image folder). correct me if I am wrong. Thanks.

Dataset file structure

Why do I run prepare_data.py and the output is two empty folders, the author, can you provide the folder structure?

关于测试集的ground truth

您好~感谢您提供的代码和数据集，请问测试集有对应的标注吗？因为最近我在调试AP和AR的评估代码没有找到truth_dir的路径，另外您最新上传的convert_rodnet_to_rod2021.py是为了将test.py生成的.txt格式与标注格式相同吗？因为我看之前生成的与训练中的anno文件顺序不太一样。
期待您的回复！

checkpoint文件在哪下载？

the annotations file of the test set

Hi,
Can you share the annotations file of the test set? Thanks

`batch_size>1` is not working in the testing phase

Hi，
Have you ever tried muti-gpus training? I simply add DataParallel but the AP and AR are lower than the training with single gpu.

Thanks!

Ladar data set

Hello！I opened one of them There are many groups of data in the NPY file. Convert NPY files into pictures to get vertical lines.I would like to ask what is the meaning of the data in the radar data set.

I didn't find the camera annotation files

Hello ,I didn't find the camera annotation files,I just found the files contain annotations of the radar frames only for the radar frames. please can you help me to get the camera annotations?Thank you so much.

Questions about the format of the dataset

I have downloaded the dataset from your website. The format is shown below.

In file cruw_devkit_tutorial_rod2021.ipynb, the format is

Do I need to transfer the downloaded dataset into the format in the ipynb file?
Is the transfer script already provided?

prepare_dataset for test

I read the closed issues about prepare_dataset.py on the testset (fails because there are no image_paths), but even if you ignore that (you can't just comment it...), how can you run the test.py? It also looks for the image_paths and breaks.

Should I rewrite it all? or is there a small and smart fix?

thanks

CRDataset.py ignores trailling data

The CRDataset will not include some data files at the back of a training folder.
Say Folder 2019_04_30_MLMS001 has 900 frames, with a win_size of 16 and stride_size of 8, the CRDataset only gives a length of 111, and the last data it includes corresponds to frame 895.

if __name__ == "__main__":
    args = parse_args()
    args.config = 'configs/config_demo.py'
    config_dict = load_configs_from_file(args.config)
    dataset = CRUW(data_root="../ROD2021", sensor_config_name='sensor_config_rod2021')
    crdata_train = CRDataset(data_dir='data/', dataset=dataset, config_dict=config_dict, split='test',
                                 noise_channel=args.use_noise_channel)
    print(len(crdata_train))


    dataloader = DataLoader(crdata_train, 1, shuffle=False, num_workers=0, collate_fn=cr_collate)
    for iter, data_dict in enumerate(dataloader):
        #print(iter)
        if iter == len(dataloader) - 1:
            print(len(data_dict['image_paths'][0]))
            print(data_dict['image_paths'][0])`

outputs:

111
['../ROD2021/sequences/test/2019_04_30_MLMS001/IMAGES_0/0000000880.jpg', '../ROD2021/sequences/test/2019_04_30_MLMS001/IMAGES_0/0000000881.jpg', '../ROD2021/sequences/test/2019_04_30_MLMS001/IMAGES_0/0000000882.jpg', '../ROD2021/sequences/test/2019_04_30_MLMS001/IMAGES_0/0000000883.jpg', '../ROD2021/sequences/test/2019_04_30_MLMS001/IMAGES_0/0000000884.jpg', '../ROD2021/sequences/test/2019_04_30_MLMS001/IMAGES_0/0000000885.jpg', '../ROD2021/sequences/test/2019_04_30_MLMS001/IMAGES_0/0000000886.jpg', '../ROD2021/sequences/test/2019_04_30_MLMS001/IMAGES_0/0000000887.jpg', '../ROD2021/sequences/test/2019_04_30_MLMS001/IMAGES_0/0000000888.jpg', '../ROD2021/sequences/test/2019_04_30_MLMS001/IMAGES_0/0000000889.jpg', '../ROD2021/sequences/test/2019_04_30_MLMS001/IMAGES_0/0000000890.jpg', '../ROD2021/sequences/test/2019_04_30_MLMS001/IMAGES_0/0000000891.jpg', '../ROD2021/sequences/test/2019_04_30_MLMS001/IMAGES_0/0000000892.jpg', '../ROD2021/sequences/test/2019_04_30_MLMS001/IMAGES_0/0000000893.jpg', '../ROD2021/sequences/test/2019_04_30_MLMS001/IMAGES_0/0000000894.jpg', '../ROD2021/sequences/test/2019_04_30_MLMS001/IMAGES_0/0000000895.jpg']

Questions about confmap

For ConfMaps generated from CRF annotation:
confmap_sigmas {'pedestrian': 15, 'cyclist': 20, 'car': 30}; confmap_sigmas_interval {'pedestrian': [5, 15], 'cyclist': [8, 20], 'car': [10, 30]}; confmap_length {'pedestrian': 1, 'cyclist': 2, 'car': 3}
What do these items mean and why to choose these values? I read [48] but did not get a clue. Thanks.

Questions about radar data files in the dataset

Hi,
Can you introduce the radar data in the dataset? For example, what does the 2 in the shape 1281282 in a single npy file mean? What does it mean that the radar data corresponding to each image is 4 npy files? Thanks for your answer

关于训练模型的问题

Traceback (most recent call last):
File "tools/train.py", line 181, in
data = data_dict['radar_data']
KeyError: 'radar_data'

Can you provide the dataset for the Challenge?

I found that the dataset for the challenge and the dataset now on the website is different?

Can you provide the last dataset for the challenge?

Question with images in test data set

Hi yizhou,

thanks for your code sharing of RODNet! I have a small question about test data set: I reorganize the data files as you showed the data file structure in CRUW devkit, but I got an error during running 'perpare_data.py', because it cannot find image file in test subdir. But I thought in test stage there will only be radar data and no image file. Could you help me to make this clear? Thanks!

Access to the validation script

Hi Yizhou.

I would like to train a RODNet from scratch. To do so, it could be great to have validation scores to select the best models and analyse the training process. Referring to your paper, I'm talking about the method you used to compute the AP and AR scores using the OLS thresholds between the selected points from the predicted ConfMaps and the ground truth.
Your training script mentions a validation function that seems to have this purpose here:

RODNet/tools/train.py

Line 232 in 969cad6

# validate()

Maybe I have missed something in your code to compute the performances of the model.
If yes,, could you indicate me where I can find this information. If not, could you please integrate this function in the repo?

Thanks for your help, have a great day.
Best.
Arthur

The location of implementation for CRF algorithm

In your paper, you mention that CRF Annotations are generated from both RGB images and RAMaps Snippet by CRF algorithm. However, after reading through the code, I couldn't find where the camera image is used. It seems that the confmap is only generated by the radar data.

It would be of great help if you can help point out where the CRF algorithm is implemented. Thanks in advance.

Camera-Radar Fusion code sharing

Hi Yizhou and RODNet team members,

Thanks for the great work and sharing.

Is it possible to share the camera-radar fusion part to generate the annotation?

radar

Camera Annotations

Hi,

Thanks for the great work. It seems that the annotation files contain annotations only for the radar frames. Will it be possible to provide the camera annotations as well, or a code to map the radar annotations onto camera images to get camera annotations ?

Best Regards,
Mujtaba

How train your own data set?

How to make your own CFR annotation?The paper did not mention how to make a data set，but directly uses the CRUW data set with CFR annotations for training.

Definition of radar data

Hi Yizhou,

Thanks for the precious radar, camera and ground truth data for training.
I didn't find the definition of how the radar data(.npy in RADAR_RA_H) produced.
Could you please give a brief work flow from the raw adc data to it?
For example:
adc data --> range fft --> doppler fft --> angle fft --> save to .npy file
What are the number of Tx/Rx antenna used, number of chirp, number of range/ doppler bin in raw data?

Data set annotation problems

Hi wang,i find in your CRUW dataset have some label problems, so how do you train your model? I list a few wrong labels below

about images in test datasets

Hi，Yizhou,

Thank you for providing data to train and test
When I was reading the paper, I found that it tested radar data with some poorly lighted (nighttime) images. But when I downloaded CRUW, I found that there is no such data.I think it is probably in the test set, however there are no visual images to prove my conjecture. Recently I want to do some visual and radar fusion tasks and need these night images very much. I wonder if you can share this part of the data.

Once again, thank you sincerely!

running prepare_data.py can not get .pkl,run without error ,can you help me to solve this question?

The result of running is
"preparing train set ...
preparing test set ..."
but no .pkl is generated.

About prepare.dataset.py --configs file

HI！
When I run tools/prepare_dataset/prepare_data.py
--config configs/config_rodnet_cdc_win16.py \
There is always an error：in the load_configs.py import_module.No module name config_rodnet_cdc_win16。
It is confusing whether the configs file should not be set to this. @yizhou-wang I sincerely hope you can help me。

configuration file

--config configs/<CONFIG_FILE> What is the configuration file in the command, I have tried many times and got an error, I would like to ask for the solution!

Inference on my own dataset

Nice work, I wonder how to run RODNet on my own raw radar data (inference mode). Is it correct that I need first to generate confmap for the raw data without the classes of the objects?

Thanks!

about load_txt.py

您好，最近我在尝试通过代码和之前您在CRUW问题5中提供的网址理解AP的计算，但是看到read_rodnet_res函数第133行这里不太理解，为什么设置if agl > math.radians(60) or agl < math.radians(-60):，这是根据测试集的先验信息设置的吗？ 提前感谢您的回答！

Cannot run test.py file

When I run the test.py file, it gives the following error.
Traceback (most recent call last):
File "tools/test.py", line 165, in
image_paths = data_dict['image_paths'][0]
KeyError: 'image_paths'
As far as I know, during the testing phase, there should be no need for images. And I have deleted the image path when generating the test/.pkl file.
Has anyone met this problem?
Thank you very much for your reply.

can't create the test.pkl

After reviewing previous questions and answers, I tried to modify prepare.py, but test.pkl still cannot be generated .Can you help me? pls...

缺少cruw.annotation文件

我在运行官方给的baseline时，发现您prepare_data.py代码里有from cruw.annotation.init_json import init_meta_json这一行代码，但是在您给的cruw文件里并没有发现cruw.annotation文件

关于test.py的疑惑

您好，非常感谢您的指导
当我运行test.py的时候，我有一些疑惑。
首先，我已经在prepare_date.py程序运行过程中，成功生成了训练集中RGB图像+射频图像的.pkl文件，并且生成sequences/text/文件中射频图像的.pkl文件。

但是当我运行test.py程序过程中，我的程序停止在170行左右 data = data_dict['radar_data'] image_paths = data_dict['image_paths'][0] seq_name = data_dict['seq_names'][0] 。data_dict这个字典中并没有所需要的key。

此外，检查前文中加载的测试文件.pkl中，并没有发现关于image_paths的属性。考虑是因为生成测试.pkl文件中，并没有引用RGB文件的路径。请问我的理解是否是正确的？

我所加载config文件为：config_rodnet_cdc_win16.py
sensor_config文件为：sensor_config_rod2021
data_dir文件为：prepare_date.py中生成的射频图像的.pkl。
checkpoint文件为：训练过程中最后一轮生成的.pkl文件

感谢您提供的宝贵意见！！

About 70% of the test data download problem

Hi,I can't find the download link for the 70% test data. Can you share the download link?

ConfMap generation details unclear

Hi yizhou,

I'm reading through your code. But I'm confused about the details of confMap generation process. At this line of code, can you explain how sigma is calculated? What's the meaning of confmap_length[class_name] and range_grid[rng_idx]?

RODNet/rodnet/core/confidence_map/generate.py

Line 36 in 969cad6

sigma = 2 * np.arctan(confmap_length[class_name] / (2 * range_grid[rng_idx])) * confmap_sigmas[class_name]

Also, why you time 2 to (rng_idx - i) at the following line?

RODNet/rodnet/core/confidence_map/generate.py

Line 44 in 0505a4d

distant = (((rng_idx - i) * 2) ** 2 + (agl_idx - j) ** 2) / sigma ** 2

Thanks!

Error on radar_data

Traceback (most recent call last):
File "tools/train.py", line 199, in
loss_confmap.backward()
File "C:\anaconda3\envs\rodnet\lib\site-packages\torch\tensor.py", line 221, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "C:\anaconda3\envs\rodnet\lib\site-packages\torch\autograd_init_.py", line 130, in backward
Variable._execution_engine.run_backward(
RuntimeError: Unable to find a valid cuDNN algorithm to run convolution

about the data input of the RODNet model

Hi,
I want to test the model on a customer datasets, which is the range, azimuth and reflection(rcs) value. And i have trained the RODNet model based on the CRUW dataset . So how can I use my own datasets to test the RODNet model ?
Thanks for you help.