opendrivelab / tcp Goto Github PK
View Code? Open in Web Editor NEW[NeurIPS 2022] Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline.
License: Apache License 2.0
[NeurIPS 2022] Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline.
License: Apache License 2.0
thanks for your great work,which taught me a lot
however,I am a little confused about couple details about the network,looking for help.
1 You mentioned that you didnt use the last layer of the resnet34 which has 1000 output.but I find it in the code model.py
"self.join_traj = nn.Sequential(
nn.Linear(128+1000, 512),
nn.ReLU(inplace=True),
nn.Linear(512, 512),
nn.ReLU(inplace=True),
nn.Linear(512, 256),
nn.ReLU(inplace=True),
)"
I am not sure about how this 1000 come from
2 it is about the attention map
"self.wp_att = nn.Sequential(
nn.Linear(256+256, 256),
nn.ReLU(inplace=True),
nn.Linear(256, 298),
nn.Softmax(1)
)"
I just start study the attention machanism ,would you please show me what 298 represent?
looking forward to your reply : )
Hi,
thank you for releasing the code and dataset.
Will you also release the pretrained weights of your model that you have used for your leaderboard submission?
This would make it easier for people to reproduce the results/evaluate TCP on other benchmarks.
Hello, may I ask if there is an error in the code in tools/gen_ In data.py, there are:
full_seq_x.append(measurement['y'])
full_seq_y.append(measurement['x'])
The correct one should be the following:
full_seq_x.append(measurement['x'])
full_seq_y.append(measurement['y'])
right?
Sorry for the qusetion, I have downloaded all the datasets from Baidu. But I do not know how to set the dataset path in TCP/config.py
.
I am using window os. Thank you
Dear All,
I am a student engaged in automatic driving project. I'm currently running your code. Please provide us with your model ckpt file and results_ TCP.json file can be sent to us by Baidu Cloud if possible. Thank you.
thank you
I am curious about a question, why not use an official expert agent to collect data(auto_pilot.py), but rather use a roach (roach_ap_agent.py)
Thanks for your amazing work!
when I try to generate dataset there is a error
(TCP) hxa@hxa-Nitro-AN515-58:~/TCP$ sh leaderboard/scripts/data_collection.sh
init statistics_manager
begin
========= Preparing RouteScenario_0 (repetition 0) =========
> Setting up the agent
> Loading the world
Town01
load
load
load
The scenario could not be loaded:
> set_global_plan() missing 1 required positional argument: 'wp_route'
Traceback (most recent call last):
File "leaderboard/leaderboard/leaderboard_evaluator.py", line 352, in _load_and_run_scenario
scenario = RouteScenario(world=self.world, config=config, debug_mode=args.debug)
File "/home/hxa/InterFuser/leaderboard/leaderboard/scenarios/route_scenario.py", line 189, in __init__
self._update_route(world, config, debug_mode>0)
File "/home/hxa/InterFuser/leaderboard/leaderboard/scenarios/route_scenario.py", line 229, in _update_route
config.agent.set_global_plan(gps_route, self.route)
TypeError: set_global_plan() missing 1 required positional argument: 'wp_route'
> Registering the route statistics
Exception ignored in: <function RouteScenario.__del__ at 0x7f15ef125950>
Traceback (most recent call last):
File "/home/hxa/InterFuser/leaderboard/leaderboard/scenarios/route_scenario.py", line 572, in __del__
self.remove_all_actors()
File "/home/hxa/InterFuser/scenario_runner/srunner/scenarios/basic_scenario.py", line 202, in remove_all_actors
for i, _ in enumerate(self.other_actors):
AttributeError: 'RouteScenario' object has no attribute 'other_actors'
Could you please tell me how to deal with it?
By the way, the trainning and evaluation are both running properly.
Hello, I was going through the data generation pipeline and noticed that the x and y coordinates are switched around in the data packing stage.
Is this a typo, or could you provide me with some rational for why this is necessary please? Thank you!
I have tried multiple times trying to download dataset provided in google drive. It proceed for some time but it is cancelled due to "network error". Also i tried to download from baidu link which is mentioned in the issue (#12), but cannot reach the site. Could you share another link for dataset?
In the paper, it is mentioned that "data samples are stored at 2Hz" (appendix A) and "future steps at 2Hz are predicted for both the trajectory branch and the control branch" (appendix B).
However, from the code, it seems that the control frequency is 20Hz:
https://github.com/OpenPerceptionX/TCP/blob/876cca2042009601e510ffe2b043c43b09ccbf35/leaderboard/leaderboard/leaderboard_evaluator.py#L90
Does this mean that we deploy the trained model with a higher frame rate? Or did I miss something?
Thanks!
Hi,when I run the approach_ The agent collected data and reported this error:
AttributeError: 'NoneType' object has no attribute 'get_location'
leaderboard.autoagents.agent_wrapper.AgentError: 'NoneType' object has no attribute 'get_location'
Why?
Hello,
I was very interested in the fact that you combined both branches using a dynamic alpha value, but it seems in tcp_agent.py (located in leaderboard/team_code) it seems that alpha is set to 0.3 and never changed? I would like to know if I'm missing something regarding where the fusion is happening as I would like to understand the code for this better.
Thank you.
May I ask how many training epochs did you have when submitting the leaderboard list? I think the code is set to 60?
May I ask where the time limit for agent blocking is configured in the code?
Hi! Really cool project. And thanks for sharing the pre-trained models.
When running the default evaluation script (adding new.ckpt in TEAM_CONFIG), no new Pygame window appears with ego's front view.
And I couldn't find any display interface in the tcp_agent code either.
All I see is the bird's view in the main Carla window, with ego following the route.
Am I missing something?
Thanks!
May I ask why the image size is defined as 900X256? Isn't the field of view of the image the same?
Dear All,
I am a student working on a project involving testing ADAS/AV and scenario synthesis using BeamNG.tech and would love to run (test) your driving agent in that simulator. I know CARLA is kind of a de facto standard, but IMHO BeamNG.tech is superior when it comes to physic simulation, content, and flexibility. Further, BeamNG.tech is free for research, offers a python API, just like CARLA, and implements a wide range of sensors.
So I wonder how technically difficult it would be to port TCP to BeamNG.tech and whether anyone of you could support me (and my colleagues) in doing so. Hope to hear from you soon,
Thanks!
-- Benedikt Steininger
Hi~ author, In my opinion, TCP model directly use raw image and some measurement signal as input, and doesn't consider intermediate perception results. But how does it learn traffic light information? If only rely on expert trajectory samples to train, I think the traffic light is too small in front view such that it's actually hard to learn "red-stop, green-start" behavior?
Besides, does training dataset size has crucial impact on the final performance of understanding traffic light? Whether there are relevant ablation experiments about this?
While running the evaluation, I noticed that the tcp_agent.py
dumps the network predictions in a json file. Is is possible to project these waypoint coordinates on the image?
Thanks!
Thanks for your great work. When will the code be released?
In your paper, TCP-SB replaces shared encoders of TCP with two sparate ones for two branches. But I don't understand this shared encoders . Can you help me explain which part of the corresponding code?
When running tools/filter_data.py, the path cannot be found:
FileNotFoundError: [Errno 2] No such file or directory: 'routes_town05_long.json'
May I ask if this file will be generated when collecting data? I don't seem to have found the relevant generation code?
Thanks for your great work. I'd like to konw when will the code be released?
Hello, may I ask which xml file is the score for uploading the leaderboard list? is: leaderboard/data/official/routes_testing.xml? right?
It said that it collected 8 towns' data, but in the CARLA leaderboard, it just provided the six towns' routes XML. Paper said, "it is generated randomly with length ranging from 50 meters to 300 meters."
Could you please give the route files that the data collected or the random scripts? I think the training data sometimes is more important to the model... just sometimes. I'm just curious about the exact detail of the data routes. Since there is no table about others' methods but trained on your dataset in the paper also.
The detail about the measurement encoder didn't illustrate in the paper, is that just one MLP like concat[v, turn_left, (x,y)] -> to the desired size? maybe adding the network details about the output, and input size will help readers know better about the network detail.
Or what's the exact output size on
the input about the whole network is just one image or
Hey, thanks for such an amazing work! Do you have any timelines by when the data would be released - and if possible, what format it might be in? Cheers!
你好,我对训练时的数据处理有疑惑,可以帮忙解答一下吗?
按照论文的思路,是要输入连续帧的图像进行训练吗?因为同一个场景连续帧数据的存储记忆训练(GRU)才有意义?如果是的话,要怎么样输入训练数据呢?因为数据包括了多个城镇和多条路线,存储的连续帧信息不一致。
Hello,
I am inquiring to confirm whether the dataset available on the provided Google Drive link was used for training the model that achieved the top position on the CARLA leaderboard. This verification is crucial for understanding the dataset's relevance to the leaderboard results.
Your response will greatly assist me.
Thank you for your attention to this matter.
When submitting the leaderboard, the training dataset was for 8 towns. What is the val dataset?
hello! first thank you for releasing this codebase. though it might be a silly problem, but i want to ask the environment that you setup the carla program: in a linux server or a computer with desktop system? because i was trying to build carla in our online server, but found that it requires to reboot the computer. we can't reboot it for installing this environment.
Hello! I noticed that the Town07 scenarios and Town10 scenarios actually contain town02 scenarios. Could you update this with the correct scenarios please?
Could you also point me to how these scenarios were generated and how I can generate my own scenarios please? Thank you!
Hi everyone,
As the TCP implementation goes on, I realized that the current settings for the training (i.e., 60 epochs with LR decay after 30 epochs) result in severe overfitting to LANE FOLLOWING mode. It is normal considering the general distribution of the CARLA-based autonomous driving datasets.
Have you tried anything (e.g., weighted sampling) to solve that issue?
Hi~ author, I wanna ask if measurement state is well designed for the best metric?
For example, is command info (like "turn left") is a must? can I just use a target point (x,y) and ego speed V as ego measurement state input? because I think the target point position already contains the command info( 'turn' or 'go straight' ).
@penghao-wu
Hi, Since tcp_agent is dumping the predicted waypoints, can you please share the visualization code for projecting it on the bev image.
I can only find the Google Cloud Disk download method in markdown
Excuse me, there are some operational problems about "python TCP/train.py --gpus"
python TCP/train.py --gpus 1
Traceback (most recent call last):
File "TCP/train.py", line 171, in
train_set = CARLA_Data(root=config.root_dir_all, data_folders=config.train_data, img_aug = config.img_aug)
File "/home/zjy/TCP/TCP/TCP/data.py", line 49, in init
data = np.load(os.path.join(sub_root, "packed_data.npy"), allow_pickle=True).item()
File "/home/zjy/anaconda3/envs/TCP/lib/python3.7/site-packages/numpy/lib/npyio.py", line 416, in load
fid = stack.enter_context(open(os_fspath(file), "rb"))
FileNotFoundError: [Errno 2] No such file or directory: 'Tcp/tcp_carla_data/town01/packed_data.npy'
First thank you for sharing the source code and datasets. Really an excellent work.
However, When I following the manual to run the evaluation by sh leaderboard/scripts/run_evaluation.sh
, I found out the python process has an extremely high CPU usage (basically about 6000%).
And I also check the usage of GPU (single 4090), only 2.3G of memory usage and average 5% volatile gpu-util, basically idle. Is that mean it was just using CPU to run the model?
So I wonder if it is a normal case and is there any suggestion about how to dealing with that? Thank you!
Hi,
Thank you for releasing the code and the dataset.
Sorry for the question, but I'm having trouble running the run_evaluation.sh file. I've placed the new.ckpt file and the results_TCP.json file in the outer TCP directory. And I've just edited the run_evaluation.sh line 23 to be:
export TEAM_CONFIG= new.ckpt
This results in an error which can be traced back to the watchdog.py file which says:
"NoneType object has no attribute cancel"
I believe this is the case of the model file not being accessible.
Can you please tell me if this is the correct way to add the path to the model file? If not, how should I change it?
Thank You
Hi again :)
So far, I've reimplemented the control branch only part of the TCP. It is basically the version that you have used in the first part of your ablation study. I followed a similar procedure to generate the dataset, except it is collected with Roach code. However, the dataset size is similar to the one that you have used in your ablations.
When I conduct a benchmark with the trained model, it just gives terrible results for Town02 (DS ~ 0.02) but not for Town05 (DS ~ 0.2). However, even for Town05, I am not able to replicate the result in the paper.
Have you ever encountered a similar situation where the model performs relatively better at Town05 but not in Town02? If you have, what was your approach to solve it? Do you have any recommendations?
Thanks in advance! 💯
Hello, first of all thank you for sharing the TCP code base!
When we attempted to use TCP with Carla 0.9.13, we found that the TCP agent returned "break" value is almost always greater than zero. Due to that the Ego car is often struck and does not move. Please share if there is something I need to do to fix this.
Thanks,
Bala
Hi,
I would greatly appreciate it if you could share the code that you used for the visualization in figures 2 and 3 of the supplementary material? (trajectory-guided attention maps, GradCam and EigenCam)
Thank you very much in advance!
Hi,
First, I would like to congratulate you on this great work.
Currently, I am trying to replicate your work. Assuming that the whole network is being trained altogether (no module freezing etc.), it takes a lot of time to train the model due to the limited capacity of GPUs. Therefore, one reasonable option seems to freeze the image and measurement encoder part during the training of the auto-regressive trajectory and multi-step control prediction module. Did you do something like this? What was your approach? Do you have any suggestions?
Also, when will the code be released?
Best.
Hi again 😄
I have a few questions regarding the waypoint generation and loss calculation.
First of all, there are a lot of feature vector representations in the paper (i.e.,
hidden = GRU([control_action, j], hidden) # first hidden is zeros(256)
j = FeatureEncoder(hidden)
control_action = PolicyHead(j)
Waypoint Prediction Part:
hidden = GRU([waypoint, j], hidden) # first hidden is zeros(256), first waypoint is [0, 0]
j = FeatureEncoder(hidden)
delta_waypoint = WaypointHead(j)
waypoint = waypoint + delta_waypoint
I'd be very happy if you could give me an idea about the degree of correctness of my assumptions above regarding your model.
Waypoint Prediction:
The question is about the generation of ground-truth waypoints. One natural solution might be to use the GNSS data in future frames. However, when the data is collected with Roach, due to the fluctuations in control actions (typical RL issue), waypoints become so noisy. I am attaching a video here. Note that in this video, the waypoints are strided (every fourth future GNSS data). When I collect the subsequent future datapoints directly, it nearly becomes a cluster of points rather than a spline-like curve that shows the future trajectory. Example of the not-strided case is here.
Finally, I am assuming that the waypoints are predicted with respect to the car's reference frame, not World's. Then, the average of amplitudes and rotations of vectors composed by the subsequent waypoints are fed to the longitudinal and lateral PID controller.
I know I am bugging you a lot 🥲 , but I think everything will be more clear for all readers through the answers to these questions.
Hi,
Congrat to this impressive work.
I have a question relating to the image feature map. In the paper, you mentioned that you used ResNet34 pretrained on ImageNet as the image encoder. Could you please provide more details about the layers that you have used? Did you remove the last global average pooling and FC layers of the ResNet backbone?
I assume that you finally encoded each input image to only one feature maps F, since later you calculated an attention map and mapped it back to this feature map F for each step. If so, you should have added some decoder layers after the last Conv. layers block of the ResNet-34, right? Please correct me if I understand it wrongly. Thanks!
Best wishes
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.