maudzung/complex-yolov4-pytorch

The PyTorch Implementation based on YOLOv4 of the paper: "Complex-YOLO: Real-time 3D Object Detection on Point Clouds"

Welcome to Nguyen Mau Dzung's GitHub 👋

complex-yolov4-pytorch's Issues

How to show outputs in forms other than BEV?

I have been trying to find references to plot 3D boxes on the pointcloud data in forms other than BEV.

Any leads for the same?

AssertionError: scalar should be 0D

I've put the figures by referring to the train.sh file.

$ python train.py --gpu_idx 0 --multiscale_training --batch_size 4 --num_workers 4
and this error occur

It seems to be a dimension problem in the np, is there any py file that needs to be modified?

I want to know the development environment.

Thank you for sharing the source
I want to know your development environment. ubuntu, cuda, cudnn etc..
Can you tell me the program versions i need to run the program?
Are you use anaconda environment?

What are the differences between single machines and tow machines?

I am currently working on this.
Single machine (node), multiple GPUs
$ python train.py --dist-url 'tcp://127.0.0.1:29500' --dist-backend 'nccl' --multiprocessing-distributed --world-size 1 --rank 0
I use a titian RTX. Is it right use this code? What are the differences between single machines and tow machines?

......................................................................................................

this error occur.....
Do i have to use single gpu?

Calibration of Camera images

Hello, the inference on my own dataset works very good. The bounding boxes in point cloud are exact. But the transformation to my camera data is not good. Where do I have to adjust the values ? In kitti_bev_utils.py I´m able to have influence on the boxes in camera image but it doesnt look good.

Ground Truth Process

Hi, i am new at Machine Learning, you have a good work. I want to ask, there a labels kitti dataset with 15 coordinate, but when we running test.py i see there are "x, y, w, l, im, re, cls_pred". Can you explain me how the process of the labels becomes like this on test.py? thank you, and i am sorry for my bad english.

Lack of backword pass in Giou module

convex_conners = torch.cat((p_cons, t_cons), dim=0)
hull = ConvexHull(convex_conners.clone().detach().cpu().numpy())  # done on cpu, just need indices output
convex_conners = convex_conners[hull.vertices]
convex_polygon = cvt_box_2_polygon(convex_conners)
convex_area = convex_polygon.area
giou_loss += 1. - (iou - (convex_area - union) / (convex_area + 1e-16))

this problem can be here: facebookresearch/detectron2#1347

How is Anchor Box been designed?

Great works!

Could you please tell me how is anchor box size been designed in this work?
I'm a liitle confused with it.

Complex-YOLOv4-Pytorch/src/config/cfg/complex_yolov4.cfg

Line 1146 in d50e2d5

anchors = 11, 15, 0, 10, 24, 0, 11, 25, 0, 23, 49, 0, 23, 55, 0, 24, 53, 0, 24, 60, 0, 27, 63, 0, 29, 74, 0

Can I use it to train more classes?

@maudzung
Here's a question that I want to train with net with more classes.(There is a task I should give the result with car,pedestrian,cyclist and truck) Can it works? Could you plz give me some advice?Thanks a lot and forgive me for my bad English.

How to run as a live feed?

Is there a way to display the output as a live continuous feed?

I wanted to make a live lidar object detector to detect pedestrians, etc. Is it possible?

Do I need to run voxelnet first？Or just run the codes under complex-yolov4-pytorch?

Overload resolution failed

Can anyone help me with the errors that come up when I try to use the test.py file (python test.py --gpu_idx 0 --pretrained_path ../checkpoints/complex_yolov4/complex_yolov4_mse_loss.pth --cfgfile ./config/cfg/complex_yolov4. cfg --show_image)?

• Can't parse 'pt1'. Sequence item with index 0 has a wrong type
• Can't parse 'pt1'. Sequence item with index 0 has a wrong type

Can i get the trained models for this model?

Hey firstly thank you for this amazing contribution, can I get the trained models, the .pth ones?

How fast is it to inference?

What is the total processing time from pre processing to post processing within inference?

intersection of rotate bounding box error

Complex-YOLOv4-Pytorch/src/utils/cal_intersection_rotated_boxes.py

Line 78 in 564e8e3

intersection_point = line.find_intersection(Line(s, t))

I think there should be limitations to the range of intersection points.

In the following case, intersection values are calculated as 400.0 even though the boxes do not intersect.

box1 = torch.tensor([100, 100, 40, 10, np.pi / 2], dtype=torch.float).cuda()
box2 = torch.tensor([200, 100, 40, 20, 0], dtype=torch.float).cuda()

Shapely- box1_area: 400.00, box2_area: 800.00, inter: 0.00, iou: 0.0000
intersection from intersection_area(): 400.0

Real time Detection

I have some unlabeled files in kitti format, so I would like to see the predictions in real time, how does it work. How to start the real time detection ?

can i run it in windows 10 desktop machine with CPU only (no GPU)

Where is dropblock implemented?

Hi, as said in README, dropblock has been implemented in this repo, but I can not find where it is implemented.

Trouble evaluating mAP

I trained the model with only pedestrians. I tested it and noticed a lot of false positives, I didn't train for too long so I was expecting this. When I run evaluate.py I get a mAP of 0. Any guesses as to why/advice?

Thanks

how to get the detected objects' coordinates?

does anybody know, how can I obtain the coordinates(x,y,z) of the detected object?

opencv-python version problem

I found a strange problem, using the latest version of opencv-python-4.3.0.36 will cause segmentation fault. I located the problem in kitti_dataloader.py at line 164 cv2.imshow().

env:

Ubuntu 18.04; Python3.6

output:

[1] 21406 segmentation fault (core dumped) python kitti_dataloader.py --output-width 608

If someone has the same problem, you can uninstall opencv-python and execute pip install opencv-python==4.2.0.34.
Thanks for the excellent work！

resume from a checkpoint

i was wondering if anyone knows how to resume the training after 20 epochs for example, do we use the --pretrained_path and insert the model or the --resume_path

How to train with a smaller kitti dataset?

Say if I want to train with 1000 bin files, what modifications should be done in the code?

• I tried changing this, did now work.

• I also changed the train.txt file, which did not help

What else can I try?

AP for Hard, medium and easy

Evaulate.py give f1, precision, AP and recall for classes car, pedestrian and cyclist but how to get the same for easy, medium and hard category wise within a class??

Evaluate the model on test dataset

I believe that currently evaluate.py evaluates the model on validation data.
How to evaluate on test dataset?

Doubt in density map

In the paper formula used for density map is as follows:
zr (Sj ) = min (1.0, log(N + 1)/64)

But in the code it is:

normalizedCounts = np.minimum(1.0, np.log(counts + 1) / np.log(64))

Are both same? Shouldn't it be like this?:

normalizedCounts = np.minimum(1.0, np.log((counts + 1) / 64))

pre-train model

@maudzung can you share the pre-trained model ??

TensorRT conversion

@maudzung
Hi，Maudzung, nice work! I'm trying to use TensorRT to speed up the inference. Do you have some scripts that transfer the Complex-YOLOv4-Pytorch from pytorch(pth file) to TensorRT?

Thanks!

test.py

When I run the test. py file, there is no result output, and the code does not stop? What is the reason?

Results output

How can I extract the resulting predicted files of test.py ? Like the coordinates of the bounding box etc.

If I change the resolution of bev, what other parameters do I need to change？

cuda out of memory

hi, i was wondering if anyone knows how to solve this cuda out of memory problem, i've tried everything, lowering the batch size to 1, lowering the epoch number, nothing works, but at the same time i've tried other programs and they worked fine with cuda and with even normal batch=2, so i don't what to do next if anyone have any solution please help

Can't parse 'pt1'. Sequence item with index 0 has a wrong type

Bug in src/data_process/kitti_bev_utils.py; function drawRotatedBox(); line 168

I had to cast every one of these values from "corners_int" variable to int as there were actually floats. cv2.line() then raised an Exception.

cv2.line(img, (corners_int[0, 0], corners_int[0, 1]), (corners_int[3, 0], corners_int[3, 1]), (255, 255, 0), 2)
changed to:
cv2.line(img, (int(corners_int[0, 0]), int(corners_int[0, 1])), (int(corners_int[3, 0]), int(corners_int[3, 1])), (255, 255, 0), 2)

This error occurred whenever I tried to execute:
python kitti_dataloader.py --show-train-data --cutout_prob 1. --cutout_nholes 1 --cutout_fill_value 1. --cutout_ratio 0.3 --output-width 608

How many figures should I put in?

I tried
$ python train.py --gpu_idx 0 --multiscale_training --batch_size 128 --num_workers 0...
$ python train.py --gpu_idx 0 --multiscale_training --batch_size 128 --num_workers 1...
$ python train.py --gpu_idx 0 --multiscale_training --batch_size 128 --num_workers 16...

but train.py: error: argument --num_workers: invalid int value: '0...' this error occur
train.py: error: argument --num_workers: invalid int value: '1...' this error occur
train.py: error: argument --num_workers: invalid int value: '16...' this error occur

train.py file (command)

can someone please finish the command of the train.py file, because there is a train.py --gpu_idx 0 --batch_size --num_workers ... and i don't know what is the rest of that command or what to put instead of the N, i am still learning, thank you.

train time

how long does it take for the author to train 300 epoch?

Did you compare speed and accuracy of Complex-YOLOv4 vs other algorithms on Kitti dataset?

@maudzung Hi,
Nice work!
Did you compare speed and accuracy of Complex-YOLOv4-Pytorch vs other algorithms on Kitti dataset?
Is it still better in accuracy and speed than other competitors?

---

Also some reference with implementations of CIoU.

Examples:

• C: https://github.com/AlexeyAB/darknet/blob/a71b9a6e9a009cf94900c53deb344c5204835700/src/box.c#L233-L256

• Matlab: https://github.com/Zzh-tju/DIoU/blob/master/simulation%20experiment/dCIOU.m

• Python: https://github.com/VCasecnikovs/Yet-Another-YOLOv4-Pytorch/blob/2e18612e1852abbf35b4dac55a00f2a3b2d814ed/model.py#L527-L561

• Python: https://github.com/ultralytics/yolov3/blob/eca5b9c1d36e4f73bf2f94e141d864f1c2739e23/utils/utils.py#L262-L282

Desctiption: https://medium.com/@jonathan_hui/yolov4-c9901eaa8e61

cuda deserialization issue

Attempting to deserialize object on CUDA device 2 but torch.cuda.device_count() is 1. Please use torch.load with map_location to map your storages to an existing device.

what to do?

i am running pretrained model

Does this complex-yolov4 network support training rectangular BEV input？

@maudzung Hello, thanks for this great work. Does the complex-yolov4 network network support training rectangular BEV input ? I think the bev map input of rectangle is better than square because of the road scene.

train point cloud with 3D Yolo with single frame

I have a point cloud of a scene that contains some 3D objects in pcd, ply or bin format and without the jpg images, only the point cloud. I have created a labeling of 2 different classes using a toolbox. It is possible to train a 3D-YOLOv4 model, someone can help me with the process with a tutorial, I can pay for it, I need this part for my degree project.

Thank you.

What's the relationship between the two repositories of 'Complex-YOLOv4-Pytorch' and 'Super-Fast-Accurate-3D-Object-Detection'

test.py Runtime Error

Traceback (most recent call last):
  File "test.py", line 98, in <module>
    model.load_state_dict(torch.load(configs.pretrained_path))
  File "/home/chaejin/anaconda3/envs/complex-yolov4/lib/python3.8/site-packages/torch/serialization.py", line 593, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "/home/chaejin/anaconda3/envs/complex-yolov4/lib/python3.8/site-packages/torch/serialization.py", line 773, in _legacy_load
    result = unpickler.load()
  File "/home/chaejin/anaconda3/envs/complex-yolov4/lib/python3.8/site-packages/torch/serialization.py", line 729, in persistent_load
    deserialized_objects[root_key] = restore_location(obj, location)
  File "/home/chaejin/anaconda3/envs/complex-yolov4/lib/python3.8/site-packages/torch/serialization.py", line 178, in default_restore_location
    result = fn(storage, location)
  File "/home/chaejin/anaconda3/envs/complex-yolov4/lib/python3.8/site-packages/torch/serialization.py", line 154, in _cuda_deserialize
    device = validate_cuda_device(location)
  File "/home/chaejin/anaconda3/envs/complex-yolov4/lib/python3.8/site-packages/torch/serialization.py", line 144, in validate_cuda_device
    raise RuntimeError('Attempting to deserialize object on CUDA device '
RuntimeError: Attempting to deserialize object on CUDA device 2 but torch.cuda.device_count() is 1. Please use torch.load with map_location to map your storages to an existing device.

I got this Runtime Error.

Nvidia GPU was operating well,,
I don't know why this program was not operated
Ofcourse, i gave arg map_location = "cuda:0". result was same
Please test or review this program..

RuntimeError: Error(s) in loading state_dict for Darknet:

I succeeded in step 2.4.1.

I'm currently working on step 2.4.2.
2.4.2. Inference
python test.py --gpu_idx 0 --pretrained_path ...

In this step I think pretrained_path means complex_yolov3.pth so I made file and put complex_yolov3.pth

And run python test.py --gpu_idx 0 --pretrained_path /home/kaai/Complex-YOLOv4-Pytorch/ab/complex_yolov3.pth

Is there anything wrong with my progress?

cuda out of memory

i've tried the code on a computer with graphic card GTX 1050 Ti and i still got the Cuda out of memory error, when i tried to lowed the batch size, it worked for a moment then stopped as it is showed in the picture below

parameter modification for better results

Hello, I am new to machine learning; I wondered what parameters I should change or modify to attempt to enhance the performance of the model?

IDE

Hi all,
Which IDE are you using?
Best regads,
PeterPham

Cuda out of memory,

hello when i tried to run the evaluate.py file , like it is mentioned in the Readme file, i always have this error, and i don't know how to fix it, mind you that i have 2Gb nvidia graphic card GTX 950m, when i tried to lower the batch size it worked, but i lowered it till it was equal 1 and the results were wrong, can you please help me

Hard coded calibration parameters & using a custom dataset

Hi,
I've had some success using a custom dataset converted to KITTI format for training. Your repo is generally easy to use so thanks for that.

However I noticed that some of the hard-coded KITTI calibration matrices are being used in augmentation. The dataset loader seems to load calibration files properly(?), but some functions use the "average KITTI value" defined here https://github.com/maudzung/Complex-YOLOv4-Pytorch/blob/master/src/config/kitti_config.py . I could change those matrices but that wouldn't scale well, so I modified the code to make sure the proper calibration values were being passed to the augmentation function.

In the end, that had a small, negative impact on my results; do you have any idea why that is? Why were things even working with the hard coded KITTI params, when I'm using a dataset completely unlike KITTI? Can you explain how the hard-coded values were used and how it would impact a custom dataset?

I'm worried I somehow broke some augmentation code in the process; do you have any suggestions with debugging this repo? (making sure the boxes are TF'd properly, data augmentation is working properly, and so on..)?

Clarification about Heightmap

As per the Complex Yolo paper, in the G field of RGB map of point cloud maximum height is encoded.
zg (Sj ) = max(PΩi→j · [0, 0, 1]T )

However, it seems that in this implementation it is normalized height:

max_height = float(np.abs(bc['maxZ'] - bc['minZ']))
heightMap[np.int_(PointCloud_frac[:, 0]), np.int_(PointCloud_frac[:, 1])] = PointCloud_frac[:, 2] / max_height

Could you please clarify this?

can't run because of low Vram

RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 3.79 GiB total capacity; 2.48 GiB already allocated; 25.50 MiB free; 2.52 GiB reserved in total by PyTorch)

I tried to lower the batch_size to 1 in train_config but i still have not enough memory.
is there any parameter to change so it becomes less memory hungry?
what do you suggest?