wayveai/driving-with-llms

PyTorch implementation for the paper "Driving with LLMs: Fusing Object-Level Vector Modality for Explainable Autonomous Driving"

driving-with-llms's Issues

Pre trained model

@olegsinavski @melights @jltorresm @eltociear can u please share thr pretrained model ??

How to Use Multi-GPU Training

How to Use Multi-GPU Training

The code and training command of the stage-1: vector representation pre-training stage

Can you provide the code and training command of the first stage: vector representation pre-training stage?

You can optimkze frequent routes like soc designs

You can optimkze frequent routes like soc designs

Dataset and LLM

Hello, your work is very exciting. First of all, I would like to know if your model contains a control signal output. Does the dataset contain a control signal? Second, if I want to replace the LLM with the pre-trained model of the first stage, how can I do it?

Have you ever tested in real-world driving environment?

Have you ever tested in real-world driving environment?

About the prompt

Driving-with-LLMs/utils/training_utils.py

Line 211 in ae36cc3

{data_point["output"]}"""

Hi, codes here already integrated output label into prompt, so the model already know the real label before prediction? So what about the testing stage when the model don't know the real label?

Unspecified signals

Hi,

I see that the unspecified signals are not used for caption generation. But looks like they were still inputs for the full pipeline.

Would you be able to give the documentation about description of these unspecified signals ?

e.g.https://github.com/wayveai/Driving-with-LLMs/blob/main/utils/vector_utils.py#L111

Visualization of the results

@olegsinavski @jltorresm Thanks for sharing the code base can we have the scripts for visualization code also to validate the results

Thanks in advance

The environment

I'm grateful for your contributing work. Could you please introduce your setup like CUDA and cudnn? I have tried many times to run the code 'pip install -r requirements.txt.lock' but there are many different problems and errors.

RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

Hello, when I evaluate for Perception and Action Prediction, I got this error for decapoda-research/llama-7b-hf.
How can I fix this? Thanks!

401 Client Error: Unauthorized for url

Hi， the error ‘requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/decapoda-research/llama-7b-hf/resolve/main/config.json’ occurred wen I run train.py . Whether or not I need add my username and password in the train.py script ?