wvangansbeke/lanedetection_end2end

End-to-end Lane Detection for Self-Driving Cars (ICCV 2019 Workshop)

lanedetection_end2end's Issues

Is there a pretrained model available?

Hi. I am able to train the network on the 20% TuSimple data (~100 epochs), but the result is not very good when testing with images from other sources. I think the network need to be trained more thoroughly. Is there a pre-trained weight of the network available? Could you share it with us? Thank you!

about train

about train
could i ask some question?
tusimple dataset only include some images and *.json files .what is --image_dir and --gt_dir?(run code python main.py --image_dir /path/to/image/folder --gt_dir /path/to/ground_truth/folder --end_to_end True)

what's the β parameter？

Hi , I used the Tusimple dataset to generate the training label (the value of β) according to the description in your paper. There are several questions I would like to ask you:

1. I found that the Y coordinate generated by the pixel after the transformation matrix will produce negative coordinates?
2. Can you share several sets of generated β values?

Where do the x, y coordinates come from?

Hi,
Congrats on your great work. I am being kind of confused on a point in your paper.
You mentioned that there are two maps for the x, y coordinates, then, are they just representing the normalized pixel coordinates of every pixel in the image? e.g. The first colomn of X-map would be 0,0,0,0,0,0,0...? I don't see the point of having these maps if they are just pointing to themselves.
In this case, if you choose the first way of backprop, which is to update the coordinates, then do you mean that only the values in the X, Y-features maps are updating?
One last point, is there any correlation between the pixel and the location they are pointing to? Such as they might be pointing to a pixel of the same lane line?

about Curve_parameters.json

Hi @wvangansbeke , Could you please introduce how to generate curve parameter? Thanks.

Curve_parameters.json

Hello, I am experimenting with your model, but the result is not good, I guess there is a problem with the curve parameters that I generated from my own dataset. So I first use the M-inv in your code to make a perspective transformation of the TuSimple data and then get the curve parameters, but unlike the file Curve_parameters.json, can you tell me how to generate the details of the curve parameters?

Training error in the conventional pipeline (not end2end)

Hi,

first of all, thanks for making this project public, some good work and ideas here! To my issue, I'm trying to run the model in the conventional mode (not end2end) with:

python3 -W ignore main.py --image_dir /mount/ssd_middle/TuSimple_subset/images/ --gt_dir /mount/ssd_middle/TuSimple_subset/ground_truth/ --end_to_end False --nclasses 4

but get the following error message:

0it [00:00, ?it/s]Batch with idx [2236 879 2282 2079 262 1271 2015 1018] skipped due to singular matrix The size of tensor a (4) must match the size of tensor b (8) at non-singleton dimension 0

commenting out the try / catch block I'm getting the following error:

RuntimeError: The size of tensor a (4) must match the size of tensor b (8) at non-singleton dimension 0

does anybody know what might be the reason for this? Is any quick fix possible? For the moment, I'm interested in getting segmentation maps for lanes without modeling.

Many thanks in advance!

Best,
Alexey

Why is the weight matrix a diagonal matrix?

Lane detection in LIDAR

Hi

I would need to detect lanes and borders from a dataset which is LIDAR only. Is it possible ?

Thanks

ENet-Label-Torch is available now (a light-weight and effective lane detection model)

Our ENet-Label-Torch has been released. More details can be found in my repo.

Key features:

(1) ENet-label is a light-weight lane detection model based on ENet and adopts self attention distillation (more details can be found in our paper which will be published soon).

(2) It has 20 × fewer parameters and runs 10 × faster compared to the state-of-the-art SCNN, and achieves 72.0 (F1-measure) on CULane testing set (better than SCNN which achieves 71.6).

(Do not hesitate to try our model!!!)

Performance on CULane testing set (F1-measure):

Category	SCNN-Torch	SCNN-Tensorflow	ENet-Label-Torch
Normal	90.6	90.2	90.7
Crowded	69.7	71.9	70.8
Night	66.1	64.6	65.9
No line	43.4	45.8	44.7
Shadow	66.9	73.8	70.6
Arrow	84.1	83.8	85.8
Dazzle light	58.5	59.5	64.4
Curve	64.4	63.4	65.4
Crossroad	1990	4137	2729
Total	71.6	71.3	72.0
Runtime(ms)	133.5	--	13.4
Parameter(M)	20.72	--	0.98

Question about Normalized M to scaledup M conversion

Hello, in the source code, function test_projective_transform tranforms normalized M to image size M. I cannot understand why M_scaledup is set like that.

I'm asking why M_scaledup[1, 0] and M_scaledup[2, 0] are both 0? I think it ought to be: M_scaledup[1, 0] = M[1, 0] / 2 and M_scaledup[2, 0] = M[2, 0] / (2*resize-1)

Here's my deduction. Perspective transform does following things:

(where a_{ij} is item of normalized M)

So transforming normalized M to image size M basically does following things:

Thus M_scaledup should be:

So every item matches your code except M_scaledup[1, 0] and M_scaledup[2, 0].

Single Image Inference

Hello,

I am trying to perform inference in a single image as well.

Here's what I got; it's not working yet.

### Open Image
from PIL import Image
image = Image.open("10.jpg")

 ### 1 : Load the Model
model = Net(args)
checkpoint = torch.load("model.pth.tar")
model.load_state_dict(checkpoint['state_dict'])


### 2 : Preprocessing / Transformations
w,h = image.size
totensor = transforms.ToTensor()
resize = args.resize ### 256
image = F.crop(image, h-640, 0, 640, w)
image = F.resize(image, size=(resize, 2*resize), interpolation=Image.BILINEAR)
image = totensor(image).float()

## Pass it to the model
with torch.no_grad():
        beta0, beta1, beta2, beta3, weightmap_zeros, M, output_net, outputs_line, outputs_hozizon = model(image, True)
 ### I also tried = model.forward(image, True)

And I have an error :
RuntimeError: Expected 4-dimensional input for 4-dimensional weight [13, 3, 3, 3], but got 3-dimensional input of size [3, 256, 512] instead

The pipeline should be usable by anyone who wants to test once fixed. Can anybody help ?

Is there more experiment between End2End and other lane detection methods?

Hi, @wvangansbeke

Thanks for your great work. The result seems impressive.
Well, I am wondering if you compare the End2End method with other lane detection methods, https://arxiv.org/abs/1712.06080 and https://arxiv.org/abs/1802.05591?

I want to actually use this model in my DIY self driving car

Hello,

I'm a hobbiest software dev who has only gotten into ML topics. OpenPilot is an open source project to add L2 autonomy to most cars 2017 and newer. The code repo is here:
https://github.com/commaai/openpilot

The community is currently constrained to Qualcomm based hardware due to the provided models being in a snpe dlc format. I instead bought a Nvidia Jetson Nano and many others are working on custom hardware solutions.

I realize that you are in academia and your intents are to further the field, so this may be out of your scope. But..

It would be fantastic if I could use this model in a real world application. The outputs from a model would be consumed here (which are used by Pathplanner/MPC to generate desired steering angles) https://github.com/commaai/openpilot/blob/devel/selfdrive/controls/lib/model_parser.py

An example of output from the existing model is here (check model file):
https://www.dropbox.com/sh/7ncns9yuk7lx910/AADWlF2y9nple5in6YXtv06ba?dl=0

We can ignore path, lead as they aren't currently used. Only left, right lane is important (I think).
An example:

I see that the existing model outputs are contained in the lane object, just the scale is wrong for this use case it seems.

It would mean a great deal to many users of OpenPilot to have an alternative model to use. One community member has offered $5,000 USD for a workable solution (we are rather passionate users).

I'm training a model now using your codebase and will continue tinkering to see if I can get something workable for OpenPilot. I am only a novice, so any assistance you could provide would be wonderful (although, I have no expectations).

Regardless, thank you for sharing your work!

Update: The 0.6 model puts out polys instead of X Y. I also have a network architecture diagram of their existing network (it's all OSS, just in dlc format that is unfriendly). I can collect any information you'd request if you'd take this on.

https://youtu.be/IXGizNFUFyI

How to train another dataset of multi-lane

If I train this network with another dataset, how to prepare the dataset and the code for training should be modified?

thanks.

questions about "masked"

hi Wouter:
1、in the code，Net.forward has self.net(input, end_to_end*self.pretrained)，the input image is normal image not a bird's view image?
2、the net output “masked” is showed in “save_weightmap”. I get the "wm0_zeros" , the weighted picture is not matched with the detect line.as show in the picture.
i think the left ego_lines in the wm0_zeros should be show， but cannot find left ego lines infor in the wm0_zeros.

Validation in other datasets

Have you verified in other datasets？How does the result of your paper perform in rainy days, nights and other complex weather conditions?

questions about"switching lanes" and "detect 4 lines"

hi Wouter:
in the issues i found “ I left some images out for training were the car is switching lanes, since you'll need an instance segmentation to handle those cases”
why these switching lanes cannot detect by the method.

"I used a second branch in decoder which told me which one is present in the scene" i can not find the second branch in the code? is it not released?
i can find“if self.nclasses > 3” support 4 lines.

"maximum number of lane lines is a hyperparameter"means I used n output maps in order to detect a maximum of n lane lines.(n is 5 or biger), is it right?

how to do normalization on my own dateset

Hello, I am trying to train the network on my dataset, but I do not know the details on how to do the normalization , can you please show me more? Thanks!

About the ground truth dataset?

Why the images are all black in the ground truth dataset?I downloaded it in the link provided by the author.

For training,how to organize the original Tusimple data set to get valid_index,json,gt_dir ....

run train error

hi, I tried to run you model by:
python main.py --image_dir {mypath}/tusimple/DATASET/images/ --gt_dir {mypath}/tusimple/DATASET/ground_truth/ --end_to_end True

but got the error like this:

0it [00:00, ?it/s]Batch with idx [1375  536 1493 1366  750 1664 1031 1017] skipped due to singular matrix
tensor should be 2 dimensional

The Dataset is downloaded form your link. Appropriate if you could help by this.

How to visualize single image result from model ?

Can you share file for test one image?

about paper

HI,wvangansbeke
Thanks,your good work
how do I visualize this heatmap in paper? this weight map?

How can get the transformation matrix H?

In your paper, the transformation matrix H is kownn, but I don't know how to get the matrix, is it fixed to every image in the tusimple dataset according to the camera parameters, or it's dynamic due to different image? Thanks!

Tu simple evaluation resizing

Hi,

The tusimple evaluation takes as input h samples and lane points with the size of 720 x 1280 however if we train a network on smaller size then there are two ways to go about it.

a) resize the network output image to 720 x 1280.

b) resize the points to smaller size and generate y samples/lane points pair.

Which one do you think is best and if this is considered in your code ?

pretrained weight

Hi Wouter, could you please provide me with the pre-trained model weights; or even share it? It would be a good hot-start. Thank you in advance!

Does this end-to-end lane detection algorithm work for the situation which the number of lane is more than 4?

Does this end-to-end lane detection algorithm work for the situation which the number of lane is more than 4? Thanks.

May I ask when the code is open source?

Sorry,excuse me,when will the code be released?

How to visualize the model？how to open the file of .pth.tar?

When will the code get back?

Hi, I am wondering when the license issue can be fixed so that we can check out the code.
Thanks!

How do we visualize the lane predictions similar to the clip in readme.md?

write_lsq_results' from 'Dataloader.Load_Data_new missing

sudo python3 main.py --image_dir /home/rossfisher/LaneDetection_End2End/DATASET/images --gt_dir /home/rossfisher/LaneDetection_End2End/DATASET/ground_truth --end_to_end True
Traceback (most recent call last):
  File "main.py", line 20, in <module>
    from Dataloader.Load_Data_new import get_loader, get_homography, \
ImportError: cannot import name 'write_lsq_results' from 'Dataloader.Load_Data_new' (/home/rossfisher/LaneDetection_End2End/Birds_Eye_View_Loss/Dataloader/Load_Data_new.py)

How do you perform clustering after you get a binary segmentation?

Is the final loss function derived from the lane geometry model? as shown in the paper？

how to detect all the lane not ego-lane as the readme show?

hi,
thanks for your great job, but when I read the paper and the github code, all the experiment is about ego-lane, but readme gif in the github is multi lane, I wonder how to reach this result as you, please help, thanks a lot.

How can I get the testset dataset?

Hello:
The dataset you provide in the link include 2664 images rather than 3626images. Where can I download the latest dataset?
By the way, when you do training in BackProjection_Loss, you set clas = 1 so it needs the test images, how can I get it and its' json file?
I set clas=0 so that the training can continue, but the training record has something wrong although it doesn't matter.
Best regards
Max

How to test the model on custom dataset ?

Please update the README to include instructions for testing the source code on custom dataset with pretrained weights.
Thanks and Regards,
Divye

Is code available?

Hi Wouter,

It is an impressing work, and I hope to try other ideas based on your work. I will use it in academic purpose. May I know if the code is available at this moment? If yes, could you please share the code with me? Many thanks.

My email is [email protected]

Lane detection in my own Dataset!

Hi,I am very emotional about the work you have done.I want to run the model with my own Dataset.What parameters do I need to generate the in .json file? Is it just the "poly_params" parameter used?I only generated this parameter and can run the code, but the final prediction result is not ideal. Do you have any suggestions?

end to end train

hi, Is the end2end suitable for unclassified lanes?

How to train different data sets？

Hi,
Thank you very much for your meaningful work. I've probably reproduced the results of your paper, so I want to extend this work to different data sets.
so I need your help:
How to generate the parameters of 'poly_params' and 'position'?
I made them differently from what you provided. Would you give me a detailed description?

About model on test set of tusimple

thanks you work!
I try to run model on the test set of tusimple ,I get good result!!!
but I can't show gt on image , I don't know how to get beta params of gt lane.
I tried to convert the lane coordinates of the test set to the top view, then fit the lane on the top view and get the curve parameters twice, but when I use the obtained parameters to draw the line onto the picture, I get the wrong result, may I ask? Do I need a line to normalize the coordinates and then fit?

def draw_fitted_line(img, params, resize, color=(255,0,0)):
    params = params.data.cpu().tolist()
    y_stop = 0.7
    y_prime = np.linspace(0, y_stop, 20)
    params = [0] * (4 - len(params)) + params
    d, a, b, c = [*params]
    x_pred = d*(y_prime**3) + a*(y_prime)**2 + b*(y_prime) + c
    x_pred = x_pred*(2*resize-1)
    y_prime = (1-y_prime)*(resize-1)
    lane = [(xcord, ycord) for (xcord, ycord) in zip(x_pred, y_prime)] 
    img = cv2.polylines(img, [np.int32(lane)], isClosed = False, color = color,thickness = 1)
    return img, lane

the y_prime is normalize the coordinates ? so beta parms of model output is normalize the coordinates ?
best!!
thanks

About testing precision on my own dataset

Hi, wvangansbeke:
I have transplant the pytorch code to caffe and compare their precision. The output of the network is nearly the same. But when I test the network my own dataset use my caffe code, the detection result is not ideal. What should I pay attention to when I test my own dataset? And what should I do if I want to improve the performance?

How to find the number of lanes?

Hey there, is there a way to find the number of lanes? I understand that there is a max of 4 lanes that the model can detect, but is there a way to find the number of lanes?

How to get image output

Hey Wouter,
After completing the training using
python main.py --image_dir /path/to/image/folder --gt_dir /path/to/ground_truth/folder --end_to_end True
How can do you use the weights to evaluate the model on images to get outputs similar to the ones in the readme ?

ie:

questions about label generate

Excuse me, I have several questions as below:
1. Did you generate the coefficient label on the 256x512 image or on the original 720x1280 image?
2. Why did you do the (1-y_prime) operation on y_prime when draw the lines? (in utils.py: def draw_fitted_line)

lables/label_new.json

hi wvangansbeke:
I have some questions about "label_new.json"
label_new.json
{"position": [0, 0, 1, 1, 1, 1, 1, 0], "lines": [-1, -1, 1, 0, 0, 0, 0, 1, -1, -1], "raw_file": "clips/0313-1/6040/20.jpg"}
in the Curve_parameters.json
"lanes": [[-2, -2,...,307, 299],[-2, -2,...,1250, 1265, -2, -2, -2, -2, -2],[-2, -2,...,241, 212,...-2, -2],[-2, -2,...,1229, 1269,...-2, -2], "raw_file": "clips/0313-1/6040/20.jpg", "run_time": 20, "poly_params": [[-0.00074, 0.0480, 0.466], [-0.000110, -0.020, 0.569], [-0.00282, 0.127, 0.3673], [0.00101, -0.10, 0.69]]}
question1:how to get the position [0, 0, 1, 1, 1, 1, 1, 0],
question2:how to get the lines": [-1, -1, 1, 0, 0, 0, 0, 1, -1, -1],
in the prediscuss I found "A 1 means the line is a continuous line, a 0 means it is a dashed line and a -1 means the line is not present. The latter is really important, because the network will always output 2 or 4 lane lines." but i still can not get the right mean.

Pytorch to ONNX

Hi Wouter,
Now I want to convert your model to ONNX. So I change some code in Birds_Eye_View_Loss/main.py-save_checkpoint()

def save_checkpoint(state, to_copy, epoch):
    batch_size = 1
    filepath = os.path.join(args.save_path, 'checkpoint_model_epoch_{}.pth.tar'.format(epoch))
    torch.save(state, filepath)
    dummy_input = torch.randn(batch_size, 3, args.resize, args.resize*2, requires_grad=True)
    model_onnx_path = "super_resolution.onnx"

    model = Net(args)
    model.load_state_dict(state['state_dict'])
    model.train(False)
    model.cuda()
    torch.onnx.export(model,
                      dummy_input,
                      "epoch_" + str(epoch) + model_onnx_path,
                      verbose=False
                      )
    print("onnx well done!")
...

and I input
python main.py --image_dir ... --gt_dir ... --end_to_end fasle

BUT

Traceback (most recent call last):
  File "/tmp/pycharm_project_633/venv/Scripts/pytorch2onnx.py", line 42, in <module>
    verbose=False
  File "/work/dependence/anaconda3/lib/python3.6/site-packages/torch/onnx/__init__.py", line 27, in export
    return utils.export(*args, **kwargs)
  File "/work/dependence/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 104, in export
    operator_export_type=operator_export_type)
  File "/work/dependence/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 281, in _export
    example_outputs, propagate)
  File "/work/dependence/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 224, in _model_to_graph
    graph, torch_out = _trace_and_get_graph_from_model(model, args, training)
  File "/work/dependence/anaconda3/lib/python3.6/site-packages/torch/onnx/utils.py", line 192, in _trace_and_get_graph_from_model
    trace, torch_out = torch.jit.get_trace_graph(model, args, _force_outplace=True)
  File "/work/dependence/anaconda3/lib/python3.6/site-packages/torch/jit/__init__.py", line 197, in get_trace_graph
    return LegacyTracedModule(f, _force_outplace)(*args, **kwargs)
  File "/work/dependence/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/work/dependence/anaconda3/lib/python3.6/site-packages/torch/jit/__init__.py", line 252, in forward
    out = self.inner(*trace_inputs)
  File "/work/dependence/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 487, in __call__
    result = self._slow_forward(*input, **kwargs)
  File "/work/dependence/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in _slow_forward
    result = self.forward(*input, **kwargs)
TypeError: forward() missing 1 required positional argument: 'end_to_end'

How can I input this required argument?
Thank you!
Student Lulu

Reference:https://pytorch.org/docs/master/onnx.html

AssertionError when trying Backprojection_Loss

I downloaded the linked dataset you created, then

python3 main.py --image_dir /home/rossfisher/LaneDetection_End2End/DATASET/images --gt_dir /home/rossfisher/LaneDetection_End2End/DATASET/ground_truth --end_to_end True
size train loader is 2901
size valid loader is 725
Traceback (most recent call last):
  File "main.py", line 548, in <module>
    main()
  File "main.py", line 89, in main
    split_percentage=args.split_percentage)
  File "/home/rossfisher/LaneDetection_End2End/Backprojection_Loss/Dataloader/Load_Data_new.py", line 280, in get_loader
    nclasses=nclasses)
  File "/home/rossfisher/LaneDetection_End2End/Backprojection_Loss/Dataloader/Load_Data_new.py", line 95, in __init__
    assert len(self.rgb_lst) == len(self.gt_lst) == 3626
AssertionError