opendrivelab/tcp

[NeurIPS 2022] Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline.

TCP - Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline

Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline
Penghao Wu*, Xiaosong Jia*, Li Chen*, Junchi Yan, Hongyang Li, Yu Qiao

• arXiv Paper, NeurIPS 2022
• Blog in Chinese

This repository contains the code for the paper Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline.

TCP is a simple unified framework to combine trajectory and control prediction for end-to-end autonomous driving. By time of release in June 17 2022, our method achieves new state-of-the-art on CARLA AD Leaderboard, in which we rank the first in terms of the Driving Score and Infraction Penalty using only a single camera as input.

Setup

Download and setup CARLA 0.9.10.1

mkdir carla
cd carla
wget https://carla-releases.s3.eu-west-3.amazonaws.com/Linux/CARLA_0.9.10.1.tar.gz
wget https://carla-releases.s3.eu-west-3.amazonaws.com/Linux/AdditionalMaps_0.9.10.1.tar.gz
tar -xf CARLA_0.9.10.1.tar.gz
tar -xf AdditionalMaps_0.9.10.1.tar.gz
rm CARLA_0.9.10.1.tar.gz
rm AdditionalMaps_0.9.10.1.tar.gz
cd ..

Clone this repo and build the environment

git clone https://github.com/OpenPerceptionX/TCP.git
cd TCP
conda env create -f environment.yml --name TCP
conda activate TCP

export PYTHONPATH=$PYTHONPATH:PATH_TO_TCP

Dataset

Download our dataset through GoogleDrive or BaiduYun (提取码 8174). The total size of our dataset is aroung 115G, make sure you have enough space.

Training

First, set the dataset path in TCP/config.py. Training:

python TCP/train.py --gpus NUM_OF_GPUS

Data Generation

First, launch the carla server,

cd CARLA_ROOT
./CarlaUE4.sh --world-port=2000 -opengl

Set the carla path, routes file, scenario file, and data path for data generation in leaderboard/scripts/data_collection.sh.

Start data collection

sh leaderboard/scripts/data_collection.sh

After the data collecting process, run tools/filter_data.py and tools/gen_data.py to filter out invalid data and pack the data for training.

Evaluation

First, launch the carla server,

cd CARLA_ROOT
./CarlaUE4.sh --world-port=2000 -opengl

Set the carla path, routes file, scenario file, model ckpt, and data path for evaluation in leaderboard/scripts/run_evaluation.sh.

Start the evaluation

sh leaderboard/scripts/run_evaluation.sh

Citation

If you find our repo or our paper useful, please use the following citation:

@inproceedings{wu2022trajectoryguided,
 title={Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline}, 
 author={Penghao Wu and Xiaosong Jia and Li Chen and Junchi Yan and Hongyang Li and Yu Qiao},
 booktitle={NeurIPS},
 year={2022},
}

License

All code within this repository is under Apache License 2.0.

Acknowledgements

Our code is based on several repositories:

• Transfuser
• Roach
• CARLA Leaderboard
• Scenario Runner

tcp's Issues

question about traffic light

Hi~ author, In my opinion, TCP model directly use raw image and some measurement signal as input, and doesn't consider intermediate perception results. But how does it learn traffic light information? If only rely on expert trajectory samples to train, I think the traffic light is too small in front view such that it's actually hard to learn "red-stop, green-start" behavior?

Besides, does training dataset size has crucial impact on the final performance of understanding traffic light? Whether there are relevant ablation experiments about this？

Code

I haven't understood the code for generating the action yet. Could you please explain it? Thank you

the time limit for agent blocking

May I ask where the time limit for agent blocking is configured in the code?

Is there an error with dataset generation and filtering?

Hello,
I try to generate datasets and filter them. There is no meta folder in the dataset generated using data_collection.sh. But filtering with filter_data.py requires a meta folder.
In filter_data.py

Folders generated by data_collection.sh

code

This code doesn't match the formulas in the paper. Could you please help explain,?

TCP-SB

In your paper, TCP-SB replaces shared encoders of TCP with two sparate ones for two branches. But I don't understand this shared encoders . Can you help me explain which part of the corresponding code？

training epochs

May I ask how many training epochs did you have when submitting the leaderboard list? I think the code is set to 60?

Request Test Model File

Dear All,
I am a student engaged in automatic driving project. I'm currently running your code. Please provide us with your model ckpt file and results_ TCP.json file can be sent to us by Baidu Cloud if possible. Thank you.
thank you

X and Y coordinate switched during packing data

Hello, I was going through the data generation pipeline and noticed that the x and y coordinates are switched around in the data packing stage.

https://github.com/OpenPerceptionX/TCP/blob/ddcbc046b9f53008d35143c2093cb88bcaa80a76/tools/gen_data.py#L63-L64

Is this a typo, or could you provide me with some rational for why this is necessary please? Thank you!

Visualization waypoint

@penghao-wu
Hi, Since tcp_agent is dumping the predicted waypoints, can you please share the visualization code for projecting it on the bev image.

Question on the frame rate used for data collection and control

In the paper, it is mentioned that "data samples are stored at 2Hz" (appendix A) and "future steps at 2Hz are predicted for both the trajectory branch and the control branch" (appendix B).
However, from the code, it seems that the control frequency is 20Hz:
https://github.com/OpenPerceptionX/TCP/blob/876cca2042009601e510ffe2b043c43b09ccbf35/leaderboard/leaderboard/leaderboard_evaluator.py#L90

Does this mean that we deploy the trained model with a higher frame rate? Or did I miss something?

Thanks!

High CPU usage during evaluation

First thank you for sharing the source code and datasets. Really an excellent work.

However, When I following the manual to run the evaluation by sh leaderboard/scripts/run_evaluation.sh, I found out the python process has an extremely high CPU usage (basically about 6000%).
And I also check the usage of GPU (single 4090), only 2.3G of memory usage and average 5% volatile gpu-util, basically idle. Is that mean it was just using CPU to run the model?

So I wonder if it is a normal case and is there any suggestion about how to dealing with that? Thank you!

Alpha variation

Hello,

I was very interested in the fact that you combined both branches using a dynamic alpha value, but it seems in tcp_agent.py (located in leaderboard/team_code) it seems that alpha is set to 0.3 and never changed? I would like to know if I'm missing something regarding where the fusion is happening as I would like to understand the code for this better.

Thank you.

Pygame window

Hi! Really cool project. And thanks for sharing the pre-trained models.

When running the default evaluation script (adding new.ckpt in TEAM_CONFIG), no new Pygame window appears with ego's front view.
And I couldn't find any display interface in the tcp_agent code either.
All I see is the bird's view in the main Carla window, with ego following the route.

Am I missing something?

Thanks!

rgb size

May I ask why the image size is defined as 900X256? Isn't the field of view of the image the same？

Car struck : TCP "break" value always greater than zero

Hello, first of all thank you for sharing the TCP code base!

When we attempted to use TCP with Carla 0.9.13, we found that the TCP agent returned "break" value is almost always greater than zero. Due to that the Ego car is often struck and does not move. Please share if there is something I need to do to fix this.

Thanks,
Bala

Visualizing waypoints

While running the evaluation, I noticed that the tcp_agent.py dumps the network predictions in a json file. Is is possible to project these waypoint coordinates on the image?

Thanks!

filter_data

When running tools/filter_data.py, the path cannot be found：
FileNotFoundError: [Errno 2] No such file or directory: 'routes_town05_long.json'

May I ask if this file will be generated when collecting data? I don't seem to have found the relevant generation code?

数据训练

你好，我对训练时的数据处理有疑惑，可以帮忙解答一下吗？
按照论文的思路，是要输入连续帧的图像进行训练吗？因为同一个场景连续帧数据的存储记忆训练(GRU)才有意义？如果是的话，要怎么样输入训练数据呢？因为数据包括了多个城镇和多条路线，存储的连续帧信息不一致。

Porting TCP to BeamNG.tech

Dear All,

I am a student working on a project involving testing ADAS/AV and scenario synthesis using BeamNG.tech and would love to run (test) your driving agent in that simulator. I know CARLA is kind of a de facto standard, but IMHO BeamNG.tech is superior when it comes to physic simulation, content, and flexibility. Further, BeamNG.tech is free for research, offers a python API, just like CARLA, and implements a wide range of sensors.

So I wonder how technically difficult it would be to port TCP to BeamNG.tech and whether anyone of you could support me (and my colleagues) in doing so. Hope to hear from you soon,

Thanks!

-- Benedikt Steininger

Expert data collection

I am curious about a question, why not use an official expert agent to collect data（auto_pilot.py）, but rather use a roach （roach_ap_agent.py）

Problem in Google Drive Dataset

I have tried multiple times trying to download dataset provided in google drive. It proceed for some time but it is cancelled due to "network error". Also i tried to download from baidu link which is mentioned in the issue (#12), but cannot reach the site. Could you share another link for dataset?

Excuse me, there are some operational problems about "python TCP/train.py --gpus"

Excuse me, there are some operational problems about "python TCP/train.py --gpus"

python TCP/train.py --gpus 1
Traceback (most recent call last):
File "TCP/train.py", line 171, in
train_set = CARLA_Data(root=config.root_dir_all, data_folders=config.train_data, img_aug = config.img_aug)
File "/home/zjy/TCP/TCP/TCP/data.py", line 49, in init
data = np.load(os.path.join(sub_root, "packed_data.npy"), allow_pickle=True).item()
File "/home/zjy/anaconda3/envs/TCP/lib/python3.7/site-packages/numpy/lib/npyio.py", line 416, in load
fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: 'Tcp/tcp_carla_data/town01/packed_data.npy'

Carla environment for evaluation

hello! first thank you for releasing this codebase. though it might be a silly problem, but i want to ask the environment that you setup the carla program: in a linux server or a computer with desktop system? because i was trying to build carla in our online server, but found that it requires to reboot the computer. we can't reboot it for installing this environment.

Running the evaluation

Hi,
Thank you for releasing the code and the dataset.

Sorry for the question, but I'm having trouble running the run_evaluation.sh file. I've placed the new.ckpt file and the results_TCP.json file in the outer TCP directory. And I've just edited the run_evaluation.sh line 23 to be:

export TEAM_CONFIG= new.ckpt

This results in an error which can be traced back to the watchdog.py file which says:
"NoneType object has no attribute cancel"

I believe this is the case of the model file not being accessible.

Can you please tell me if this is the correct way to add the path to the model file? If not, how should I change it?

Thank You

question about state format

Hi~ author, I wanna ask if measurement state is well designed for the best metric?
For example, is command info (like "turn left") is a must? can I just use a target point (x,y) and ego speed V as ego measurement state input? because I think the target point position already contains the command info( 'turn' or 'go straight' ).

Confirmation on Dataset

Hello,

I am inquiring to confirm whether the dataset available on the provided Google Drive link was used for training the model that achieved the top position on the CARLA leaderboard. This verification is crucial for understanding the dataset's relevance to the leaderboard results.

Your response will greatly assist me.

Thank you for your attention to this matter.

Could you please provide the script to generate town06_addition?

Thanks for your amazing work!
when I try to generate dataset, I found there is no script of town06_addition compared to your provided dataset.
could you please provide the script to generate town06_addition?

some question about the network

thanks for your great work,which taught me a lot
however,I am a little confused about couple details about the network,looking for help.
1 You mentioned that you didnt use the last layer of the resnet34 which has 1000 output.but I find it in the code model.py
"self.join_traj = nn.Sequential(
nn.Linear(128+1000, 512),
nn.ReLU(inplace=True),
nn.Linear(512, 512),
nn.ReLU(inplace=True),
nn.Linear(512, 256),
nn.ReLU(inplace=True),
)"
I am not sure about how this 1000 come from

2 it is about the attention map
"self.wp_att = nn.Sequential(
nn.Linear(256+256, 256),
nn.ReLU(inplace=True),
nn.Linear(256, 298),
nn.Softmax(1)
)"
I just start study the attention machanism ，would you please show me what 298 represent？
looking forward to your reply : )

Question about data collection

Thanks for your amazing work!
when I try to generate dataset there is a error

(TCP) hxa@hxa-Nitro-AN515-58:~/TCP$ sh leaderboard/scripts/data_collection.sh
init statistics_manager
begin

========= Preparing RouteScenario_0 (repetition 0) =========
> Setting up the agent
> Loading the world
Town01
load
load
load

The scenario could not be loaded:
> set_global_plan() missing 1 required positional argument: 'wp_route'

Traceback (most recent call last):
  File "leaderboard/leaderboard/leaderboard_evaluator.py", line 352, in _load_and_run_scenario
    scenario = RouteScenario(world=self.world, config=config, debug_mode=args.debug)
  File "/home/hxa/InterFuser/leaderboard/leaderboard/scenarios/route_scenario.py", line 189, in __init__
    self._update_route(world, config, debug_mode>0)
  File "/home/hxa/InterFuser/leaderboard/leaderboard/scenarios/route_scenario.py", line 229, in _update_route
    config.agent.set_global_plan(gps_route, self.route)
TypeError: set_global_plan() missing 1 required positional argument: 'wp_route'
> Registering the route statistics
Exception ignored in: <function RouteScenario.__del__ at 0x7f15ef125950>
Traceback (most recent call last):
  File "/home/hxa/InterFuser/leaderboard/leaderboard/scenarios/route_scenario.py", line 572, in __del__
    self.remove_all_actors()
  File "/home/hxa/InterFuser/scenario_runner/srunner/scenarios/basic_scenario.py", line 202, in remove_all_actors
    for i, _ in enumerate(self.other_actors):
AttributeError: 'RouteScenario' object has no attribute 'other_actors'

Could you please tell me how to deal with it?

By the way, the trainning and evaluation are both running properly.

Figures 2 and 3 of the supplementary material

Hi,
I would greatly appreciate it if you could share the code that you used for the visualization in figures 2 and 3 of the supplementary material? (trajectory-guided attention maps, GradCam and EigenCam)
Thank you very much in advance!

Imbalanced Dataset

Hi everyone,

As the TCP implementation goes on, I realized that the current settings for the training (i.e., 60 epochs with LR decay after 30 epochs) result in severe overfitting to LANE FOLLOWING mode. It is normal considering the general distribution of the CARLA-based autonomous driving datasets.

Have you tried anything (e.g., weighted sampling) to solve that issue?

data collection

Hi，when I run the approach_ The agent collected data and reported this error：

AttributeError: 'NoneType' object has no attribute 'get_location'
leaderboard.autoagents.agent_wrapper.AgentError: 'NoneType' object has no attribute 'get_location'

Why?

Recurrent Module Training

Hi,

First, I would like to congratulate you on this great work.
Currently, I am trying to replicate your work. Assuming that the whole network is being trained altogether (no module freezing etc.), it takes a lot of time to train the model due to the limited capacity of GPUs. Therefore, one reasonable option seems to freeze the image and measurement encoder part during the training of the auto-regressive trajectory and multi-step control prediction module. Did you do something like this? What was your approach? Do you have any suggestions?

Also, when will the code be released?

Best.

Town07 and Town10 scenarios containing Town02 scenarios

Hello! I noticed that the Town07 scenarios and Town10 scenarios actually contain town02 scenarios. Could you update this with the correct scenarios please?
Could you also point me to how these scenarios were generated and how I can generate my own scenarios please? Thank you!

some details in dataset and network

routes for training

It said that it collected 8 towns' data, but in the CARLA leaderboard, it just provided the six towns' routes XML. Paper said, "it is generated randomly with length ranging from 50 meters to 300 meters."

Could you please give the route files that the data collected or the random scripts? I think the training data sometimes is more important to the model... just sometimes. I'm just curious about the exact detail of the data routes. Since there is no table about others' methods but trained on your dataset in the paper also.

network

1. The detail about the measurement encoder didn't illustrate in the paper, is that just one MLP like concat[v, turn_left, (x,y)] -> to the desired size? maybe adding the network details about the output, and input size will help readers know better about the network detail.
   Or what's the exact output size on $\mathbf F, \bf j_m$ which formed the j^{traj}

2. the input about the whole network is just one image or $K=4$ nums of images?

When will the code be released ?

Thanks for your great work. I'd like to konw when will the code be released?

speed and steer

Hello, after obtaining the predicted waypoints, I don't quite understand the calculation formulas for speed and steering. How can I determine the acceleration between the two points? May I ask if you could help explain? Thank you!

Image Feature Map

Hi,

Congrat to this impressive work.

I have a question relating to the image feature map. In the paper, you mentioned that you used ResNet34 pretrained on ImageNet as the image encoder. Could you please provide more details about the layers that you have used? Did you remove the last global average pooling and FC layers of the ResNet backbone?

I assume that you finally encoded each input image to only one feature maps F, since later you calculated an attention map and mapped it back to this feature map F for each step. If so, you should have added some decoder layers after the last Conv. layers block of the ResNet-34, right? Please correct me if I understand it wrongly. Thanks!

Best wishes

Dataset release?

Hey, thanks for such an amazing work! Do you have any timelines by when the data would be released - and if possible, what format it might be in? Cheers!

Waypoint Generation and Loss Architecture

Hi again 😄

I have a few questions regarding the waypoint generation and loss calculation.

First of all, there are a lot of feature vector representations in the paper (i.e., $\mathbf{j_m}$, $\mathbf{j^{traj}}$, $\mathbf{j^{ctl}}$). However, I think there are some ambiguities related to them. What I assumed is the following:

1. Let's say the output of the ResNet is $\mathbf{j_{image}}$, which is the average pooled and flattened version of $\mathbf{F}$; and, the output of the measurement encoder, which accepts (high-level navigational command, speed, target velocity) triplet as input, is $\mathbf{j_m}$.
2. Concatenation of these individual feature vectors gives the $\mathbf{j} = Concat(\mathbf{j_{image}}, \mathbf{j_m})$, which then will be fed to GRUs.
3. In #2, you've mentioned that the initial hidden input of GRUs is just zeros with the same dimension of the feature vector. In this case, how are we propagating the feature information to the GRUs?
   Multi-Step Control Prediction Part:

• For the multi-step control prediction part, based on the Figure 3 in the paper, my pseudocode is as follows:
  hidden = GRU([control_action, j], hidden) # first hidden is zeros(256)
j = FeatureEncoder(hidden)
control_action = PolicyHead(j)
• Then, in the loss calculation part, $\mathbf{j_0^{ctl}}$ is just $\mathbf{j}$, and, $\mathbf{j_n^{ctl}}$ ( $1 \leq n \leq K$ ) is the intermediate feature vectors as an output of Feature Encoder .

Waypoint Prediction Part:

• For the trajectory module, in Figure 3, it does not seem like the feature vector is again fed to the input of the GRUs. However, in this case, it appears that the whole feature information is lost in the trajectory branch. Nonetheless, again, my preassumed pseudocode is as follows:
  hidden = GRU([waypoint, j], hidden) # first hidden is zeros(256), first waypoint is [0, 0]
j = FeatureEncoder(hidden)
delta_waypoint = WaypointHead(j)
waypoint = waypoint + delta_waypoint
• In the trajectory loss calculation, there is another feature prediction mentioned in the paper, which is denoted as $\mathbf{j_0^{traj}}$. Isn't it just the $\mathbf{j}$ again? Aren't we feeding the $\mathbf{j}$ to both branches?

I'd be very happy if you could give me an idea about the degree of correctness of my assumptions above regarding your model.

Waypoint Prediction:
The question is about the generation of ground-truth waypoints. One natural solution might be to use the GNSS data in future frames. However, when the data is collected with Roach, due to the fluctuations in control actions (typical RL issue), waypoints become so noisy. I am attaching a video here. Note that in this video, the waypoints are strided (every fourth future GNSS data). When I collect the subsequent future datapoints directly, it nearly becomes a cluster of points rather than a spline-like curve that shows the future trajectory. Example of the not-strided case is here.

Finally, I am assuming that the waypoints are predicted with respect to the car's reference frame, not World's. Then, the average of amplitudes and rotations of vectors composed by the subsequent waypoints are fed to the longitudinal and lateral PID controller.

I know I am bugging you a lot 🥲 , but I think everything will be more clear for all readers through the answers to these questions.

Set the dataset path to``TCP/config.py``

Sorry for the qusetion, I have downloaded all the datasets from Baidu. But I do not know how to set the dataset path in TCP/config.py.
I am using window os. Thank you

Benchmark on Town02 and Town05

Hi again :)

So far, I've reimplemented the control branch only part of the TCP. It is basically the version that you have used in the first part of your ablation study. I followed a similar procedure to generate the dataset, except it is collected with Roach code. However, the dataset size is similar to the one that you have used in your ablations.

When I conduct a benchmark with the trained model, it just gives terrible results for Town02 (DS ~ 0.02) but not for Town05 (DS ~ 0.2). However, even for Town05, I am not able to replicate the result in the paper.

Have you ever encountered a similar situation where the model performs relatively better at Town05 but not in Town02? If you have, what was your approach to solve it? Do you have any recommendations?

Thanks in advance! 💯

val dataset

When submitting the leaderboard, the training dataset was for 8 towns. What is the val dataset?

failure example

May I ask which route and scenario file test results are shown in the image of this failure example

leaderboard list

Hello, may I ask which xml file is the score for uploading the leaderboard list? is: leaderboard/data/official/routes_testing.xml? right?

when will the code be released?

Thanks for your great work. When will the code be released?

error

Hello, may I ask if there is an error in the code in tools/gen_ In data.py, there are：

full_seq_x.append(measurement['y'])
full_seq_y.append(measurement['x'])

The correct one should be the following：

full_seq_x.append(measurement['x'])
full_seq_y.append(measurement['y'])

right?

Has Baidu Cloud Link been released？

I can only find the Google Cloud Disk download method in markdown

Will the pretrained model weights for TCP be released?

Hi,

thank you for releasing the code and dataset.
Will you also release the pretrained weights of your model that you have used for your leaderboard submission?
This would make it easier for people to reproduce the results/evaluate TCP on other benchmarks.