Comments (6)
And the training time did not decrease significantly.
That depends on your cpu and io. I get about linear speed up.
There is a decline in the performance of detection compared with using a single GPU.
Please give more details how large decline, both logs(using a link or send to my email), etc..
from centerpoint.
Thank you for your reply.
That depends on your cpu and io. I get about linear speed up.
It's my mistake, the training time did decrease a lot, it has decreased from 28h to 18h. I check the GPU-Util of 4 1080Ti, and most of the time is 0. I think maybe the IO speed of my mechanical hard disk is a bottleneck although I have increased the num_workrs of DataLoader. Are you put your dataset in the SSD?
Please give more details how large decline, both logs(using a link or send to my email), etc..
My model is trained in "car".
- DistributedDataParallel:
"0.5": 0.41004305537524055,
"1.0": 0.6311094185348973,
"2.0": 0.7190380432337594,
"4.0": 0.7583501639976733 - Single GPU:
"0.5": 0.49231069012399,
"1.0": 0.6895590650747294,
"2.0": 0.7537213190378271,
"4.0": 0.7901271570637968
And logs are sent to your email.
from centerpoint.
It's my mistake, the training time did decrease a lot, it has decreased from 28h to 18h. I check the GPU-Util of 4 1080Ti, and most of the time is 0. I think maybe the IO speed of my mechanical hard disk is a bottleneck although I have increased the num_workrs of DataLoader. Are you put your dataset in the SSD?
Yeah, you can see the log. The data time is much larger than the forward time. I didn't use a SSD for training but this should definitely help a lot for your setting.
Can you tell me what specific change did you make to the code? What model is this and how did you create this subset? Regardless, it seems both dist and single training are bad for the car class. You should see something like this for car
car Nusc dist [email protected], 1.0, 2.0, 4.0
72.88, 84.48, 87.77, 89.16 mean AP: 0.8357262620298936
from centerpoint.
Well,I used PointPillars as my baseline. And the input is replaced with a BEV created manually, the backbone is replaced with a lighter 2D detection model.
from centerpoint.
I see. I can reproduce my result with 2/4/8 gpus so I don't think there is an issue with Distributed data parallel. You may need to look at other part for the discrepancy
from centerpoint.
Thank you very much, I will check my code.
from centerpoint.
Related Issues (20)
- Virtual Points Loading
- the error when i train nuscenes, i think the reason is my spconv>2.0, but how can i change the code to solve it?
- about centerpoint-pillar nuscenes test set mAP and NDS HOT 1
- Why do Waymo and NuScenes have different `us_layer_strides` even though they use the same CenterPoint?
- No such file or directory: 'data/nuScenes/infos_ Test_ 10 sweeps_ Withvelo_ Filter_ True. pkl
- Tracking bug with Hungarian
- RuntimeError: Error compiling objects for extension HOT 1
- Unable to find nuScenes devkit HOT 1
- requirements change
- Error when starting training: TypeError: string indices must be integers
- Unable to build iou3d_NMS
- When I run create_data.py use nuScenes dataset have something wrong
- What's the speed of tracker in WOD
- time
- visual pkl
- assert len(detections) == 6008
- 请问有用mini数据集得到的centerpoint的检测结果吗,类似这个infos_val_10sweeps_withvelo_filter_True.json文件
- Request for Combined Train and Validation Split File
- Personal dataset HOT 1
- Ground truth filtering
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from centerpoint.