taodeng/cdnn-traffic-saliency

CDNN: traffic driving saliency & eye tracking dataset

Paper: How Do Drivers Allocate Their Potential Attention? Driving Fixation Prediction via Convolutional Neural Networks, 2020, IEEE Transactions on Intelligent Transportation Systems

Authors: Tao Deng, Hongmei Yan, Long Qin, Thuyen Ngo, B. S. Manjunath

How Do Drivers Allocate Their Potential Attention?

Eye tracking dataset

The eye tracking dataset is released link; Password: i5q7

The folder includes: fixdata, traffic videos, traffic frames.You need to extract each frame from videos and put it in traffic frames folder.

CDNN requirements

• Pytorch 0.4.1

• Python 2.7

Contact the Author: [email protected]

Citation

@article{deng_CDNN,
author = {Deng, Tao and Yan, Hongmei and Qin, Long and Ngo, Thuyen and Manjunath, BS},
title = {How Do Drivers Allocate Their Potential Attention? Driving Fixation Prediction via Convolutional Neural Networks},
journal = {IEEE Transactions on Intelligent Transportation Systems},
ISSN = {1524-9050},
year={2020},
month={May},
volume={21},
number={5},
pages={2146-2154},
doi={10.1109/TITS.2019.2915540},
type = {Journal Article}}

cdnn-traffic-saliency's Issues

How did the. mat file come from

hello,I want to know How did the. mat file come from

how to make the fixation map

Tao,
Can u tell me the method of making the fixation map? I did not find it in this project.I want to know the detail of it.

An error is throwed out when the program runs to line 188 of main.py:

An error is throwed out when the program runs to line 188 of main.py:
Value Error: Target and input must have the same number of elements. target nelement (29491200) != input nelement (1966080)
Line 188, loss = criterion(output, target_var)

By examining the program carefully, I find that the variables of output and target_var in line 188 are different in size, such as
output nelement: (32, 1, 192, 320)), 321192320 = 1966080
target nelement: (32, 1, 720, 1280), 3217201280 = 29491200

To be fathful to the publication, I wonder how you make consistent the sizes of the variables output and target_var in line 188 of main.py?

Quantitative Evaluation Metrics

Hello,
It's a great job! Could you provide the code of performing the quantitative evaluation metrics in your paper, such as CC, SIM, EMD and KL-Div? I did not find it in this project. Thank you very much!

您好，想问下这个代码如何进行训练和测试，以及视频转换图片如何转化？

我将视频每一帧转化为img放置在traffic_frames下面，16个视频通过工具ffmpeg转化为img，使用的命令为
ffmpeg -i traffic_videos/out1.avi %d.jpg
之后发现报错找不到图片，因为没有前缀00，于是我批量将所有图片添加前缀，之后运行python main.py
运行结果如下：我不太明白哪里出了问题，有没有较为具体的如何训练和测试的步骤呢

万分感谢！！！

Model checkpoint failed

Tao,

When I try to load the checkpoint you provided using your code, it fails:

`
import torch
import torch.backends.cudnn as cudnn
from model import Model

model = Model()
model = model.cuda()
cudnn.benchmark = True

checkpoint = torch.load(file_name_best)
model.load_state_dict(checkpoint['state_dict'])
`

The log shows that torch op names differ from the defined at Model() provided in the source code:

Traceback (most recent call last): File "main.py", line 302, in <module> main() File "main.py", line 185, in main model.load_state_dict(checkpoint['state_dict']) File "/home/user/miniconda/envs/py27/lib/python2.7/site-packages/torch/nn/modules/module.py", line 719, in load_state_dict self.__class__.__name__, "\n\t".join(error_msgs))) RuntimeError: Error(s) in loading state_dict for Model: Missing key(s) in state_dict: "convd1.0.bias", "convd1.0.weight", "convd1.1.running_var", "convd1.1.bias", "convd1.1.weight", "convd1.1.running_mean", "convd1.3.bias", "convd 1.3.weight", "convd1.4.running_var", "convd1.4.bias", "convd1.4.weight", "convd1.4.running_mean", "convd2.0.bias", "convd2.0.weight", "convd2.1.running_var", "convd2.1.bias", "convd2 .1.weight", "convd2.1.running_mean", "convd2.3.bias", "convd2.3.weight", "convd2.4.running_var", "convd2.4.bias", "convd2.4.weight", "convd2.4.running_mean", "convd3.0.bias", "convd3 .0.weight", "convd3.1.running_var", "convd3.1.bias", "convd3.1.weight", "convd3.1.running_mean", "convd3.3.bias", "convd3.3.weight", "convd3.4.running_var", "convd3.4.bias", "convd3. 4.weight", "convd3.4.running_mean", "convd4.0.bias", "convd4.0.weight", "convd4.1.running_var", "convd4.1.bias", "convd4.1.weight", "convd4.1.running_mean", "convd4.3.bias", "convd4. 3.weight", "convd4.4.running_var", "convd4.4.bias", "convd4.4.weight", "convd4.4.running_mean", "convu3.0.bias", "convu3.0.weight", "convu3.1.running_var", "convu3.1.bias", "convu3.1 .weight", "convu3.1.running_mean", "convu3.3.bias", "convu3.3.weight", "convu3.4.running_var", "convu3.4.bias", "convu3.4.weight", "convu3.4.running_mean", "convu2.0.bias", "convu2.0 .weight", "convu2.1.running_var", "convu2.1.bias", "convu2.1.weight", "convu2.1.running_mean", "convu2.3.bias", "convu2.3.weight", "convu2.4.running_var", "convu2.4.bias", "convu2.4. weight", "convu2.4.running_mean", "convu1.0.bias", "convu1.0.weight", "convu1.1.running_var", "convu1.1.bias", "convu1.1.weight", "convu1.1.running_mean", "convu1.3.bias", "convu1.3. weight", "convu1.4.running_var", "convu1.4.bias", "convu1.4.weight", "convu1.4.running_mean", "convu0.bias", "convu0.weight". Unexpected key(s) in state_dict: "module.convd1.0.weight", "module.convd1.0.bias", "module.convd1.1.weight", "module.convd1.1.bias", "module.convd1.1.running_mean", "module.c onvd1.1.running_var", "module.convd1.3.weight", "module.convd1.3.bias", "module.convd1.4.weight", "module.convd1.4.bias", "module.convd1.4.running_mean", "module.convd1.4.running_var ", "module.convd2.0.weight", "module.convd2.0.bias", "module.convd2.1.weight", "module.convd2.1.bias", "module.convd2.1.running_mean", "module.convd2.1.running_var", "module.convd2.3 .weight", "module.convd2.3.bias", "module.convd2.4.weight", "module.convd2.4.bias", "module.convd2.4.running_mean", "module.convd2.4.running_var", "module.convd3.0.weight", "module.c onvd3.0.bias", "module.convd3.1.weight", "module.convd3.1.bias", "module.convd3.1.running_mean", "module.convd3.1.running_var", "module.convd3.3.weight", "module.convd3.3.bias", "mod ule.convd3.4.weight", "module.convd3.4.bias", "module.convd3.4.running_mean", "module.convd3.4.running_var", "module.convd4.0.weight", "module.convd4.0.bias", "module.convd4.1.weight ", "module.convd4.1.bias", "module.convd4.1.running_mean", "module.convd4.1.running_var", "module.convd4.3.weight", "module.convd4.3.bias", "module.convd4.4.weight", "module.convd4.4 .bias", "module.convd4.4.running_mean", "module.convd4.4.running_var", "module.convu3.0.weight", "module.convu3.0.bias", "module.convu3.1.weight", "module.convu3.1.bias", "module.con vu3.1.running_mean", "module.convu3.1.running_var", "module.convu3.3.weight", "module.convu3.3.bias", "module.convu3.4.weight", "module.convu3.4.bias", "module.convu3.4.running_mean" , "module.convu3.4.running_var", "module.convu2.0.weight", "module.convu2.0.bias", "module.convu2.1.weight", "module.convu2.1.bias", "module.convu2.1.running_mean", "module.convu2.1. running_var", "module.convu2.3.weight", "module.convu2.3.bias", "module.convu2.4.weight", "module.convu2.4.bias", "module.convu2.4.running_mean", "module.convu2.4.running_var", "modu le.convu1.0.weight", "module.convu1.0.bias", "module.convu1.1.weight", "module.convu1.1.bias", "module.convu1.1.running_mean", "module.convu1.1.running_var", "module.convu1.3.weight" , "module.convu1.3.bias", "module.convu1.4.weight", "module.convu1.4.bias", "module.convu1.4.running_mean", "module.convu1.4.running_var", "module.convu0.weight", "module.convu0.bias ".

Am I missing something? If not, Could you provide the model used in the paper which is able to be loaded by this code?

Kind regards,
Javier