linbo0518/cvbenchmarks

Benchmarks for popular classification and object detection models on CPUs and GPUs

CV Benchmarks

Benchmarks for popular classification and object detection models on CPUs and GPUs.

Pretrained model parameters provided by gluon-cv model zoo

Env Setup

• CPU: Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz
• GPU: NVIDIA TITAN XP
• RAM: 252G
• OS: Ubuntu 16.04
• CUDA: 8.0.61
• cuDNN: 5.1.5
• Python: 3.6.8
• MXNet: 1.4.0 w/o mkl
• TVM: 0.6.dev

Classification

Network Arch	Input Shape	CPU time(ms)	GPU time(ms)	VRAM(MB)	TVM CPU time(ms)	TVM GPU time(ms)	TVM VRAM(MB)	Citation
MobileNet (1.0)	1x3x224x224	48.00	4.55	477	3.10	0.92	403	[1]
MobileNet v2 (1.0)	1x3x224x224	62.06	8.53	483	3.12	1.65	407	[2]
VGG16	1x3x224x224	420.59	5.74	1577	82.82	4.23	1053	[7]
ResNet50	1x3x224x224	93.07	10.81	701	18.38	3.90	529	[3]

Detection

Network Arch (Backbone)	Input Shape	CPU time(ms)	GPU time(ms)	VRAM(MB)	TVM CPU time(ms)	TVM GPU time(ms)	TVM VRAM(MB)	Citation
Faster RCNN (ResNet50)	1x3x800x800	15480.15	371.80	2945	tvm not support yet	tvm not support yet	tvm not support yet	[4]
SSD (MobileNet (1.0))	1x3x512x512	408.30	24.29	775	300.18	tvm not support yet	tvm not support yet	[5]
SSD (ResNet50)	1x3x512x512	678.04	33.02	1065	377.31	tvm not support yet	tvm not support yet	[5]
YOLO v3 (MobileNet (1.0))	1x3x416x416	479.82	17.72	771	61.70	tvm not support yet	tvm not support yet	[6]
YOLO v3 (DarkNet53)	1x3x416x416	843.06	27.55	1109	119.26	tvm not support yet	tvm not support yet	[6]

---

Env Setup

• CPU: Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz
• GPU: GeForce GTX 1080 Ti
• RAM: 31.3G
• OS: Ubuntu 16.04
• CUDA: 10.0.130
• cuDNN: 7.5.0
• Python: 3.6.8
• MXNet: 1.4.0 w/ mkl
• TVM: 0.6.dev

Classification

Network Arch	Input Shape	CPU time(ms)	GPU time(ms)	VRAM(MB)	TVM CPU time(ms)	TVM GPU time(ms)	TVM VRAM(MB)	Citation
MobileNet (1.0)	1x3x224x224	8.30	2.22	503	5.92	0.74	407	[1]
MobileNet v2 (1.0)	1x3x224x224	18.85	3.98	505	4.29	0.89	409	[2]
VGG16	1x3x224x224	89.55	5.46	1605	178.67	4.21	1061	[7]
ResNet50	1x3x224x224	28.13	6.37	723	42.74	3.42	535	[3]

Detection

Network Arch (Backbone)	Input Shape	CPU time(ms)	GPU time(ms)	VRAM(MB)	TVM CPU time(ms)	TVM GPU time(ms)	TVM VRAM(MB)	Citation
Faster RCNN (ResNet50)	1x3x800x800	5565.82	326.32	2949	tvm not support yet	tvm not support yet	tvm not support yet	[4]
SSD (MobileNet (1.0))	1x3x512x512	95.28	15.93	777	726.48	tvm not support yet	tvm not support yet	[5]
SSD (ResNet50)	1x3x512x512	211.78	23.25	1069	941.38	tvm not support yet	tvm not support yet	[5]
YOLO v3 (MobileNet (1.0))	1x3x416x416	165.48	10.96	775	156.73	tvm not support yet	tvm not support yet	[6]
YOLO v3 (DarkNet53)	1x3x416x416	315.16	18.64	1105	359.61	tvm not support yet	tvm not support yet	[6]

---

Env Setup

• CPU: Intel Core i7-6700HQ @ 2.60GHz
• GPU: None
• RAM: 16.0G
• OS: macOS 10.14
• CUDA: None
• cuDNN: None
• Python: 3.6.8
• MXNet: 1.4.0 w/o mkl
• TVM: 0.6.dev

Classification

Network Arch	Input Shape	CPU time(ms)	TVM CPU time(ms)	Citation
MobileNet (1.0)	1x3x224x224	34.05	8.71	[1]
MobileNet v2 (1.0)	1x3x224x224	64.42	8.41	[2]
VGG16	1x3x224x224	261.23	212.48	[7]
ResNet50	1x3x224x224	80.02	67.62	[3]

Detection

Network Arch (Backbone)	Input Shape	CPU time(ms)	TVM CPU time(ms)	Citation
Faster RCNN (ResNet50)	1x3x800x800	16197.89	tvm not support yet	[4]
SSD (MobileNet (1.0))	1x3x512x512	372.46	1111.51	[5]
SSD (ResNet50)	1x3x512x512	656.17	1350.06	[5]
YOLO v3 (MobileNet (1.0))	1x3x416x416	562.84	205.64	[6]
YOLO v3 (DarkNet53)	1x3x416x416	841.82	533.84	[6]

---

P.S. the TVM CPU time of SSD should be caused by some kind of bug or mistake, the result is really the case.

---

cvbenchmarks's Issues

A very, very strange result on MobileNet

I have recently used your code to evaluate mobilenet. My results show that GPU's performance is not as good as CPU's.

Here are my environments.

mxnet == 1.7.0
Ununtu 20.1
CUDA  10.2
Nvidia GTX 1070 Ti

Here my CPU results.

start mobilenet1.0 speed benchmark
mxnet: 0 time 5.468854904174805
mxnet: 1 time 5.232152938842773
mxnet: 2 time 5.269920825958252
mxnet: 3 time 5.603036880493164
mxnet: 4 time 5.499334335327148
mxnet: 5 time 5.205817222595215
mxnet: 6 time 5.577330589294434
mxnet: 7 time 5.659034252166748
mxnet: 8 time 5.7265305519104
mxnet: 9 time 6.112978458404541
mxnet: mobilenet1.0 5.205774307250977 5.535458087921143 6.112930774688721

Here my GPU results.

start mobilenet1.0 speed benchmark
mxnet: 0 time 40.76141119003296
mxnet: 1 time 40.58210611343384
mxnet: 2 time 41.749138832092285
mxnet: 3 time 40.48752307891846
mxnet: 4 time 40.23917198181152
mxnet: 5 time 40.33033847808838
mxnet: 6 time 40.38513660430908
mxnet: 7 time 40.24947166442871
mxnet: 8 time 40.08643388748169
mxnet: 9 time 39.98572111129761
mxnet: mobilenet1.0 39.98570203781128 40.48562288284302 41.74911975860596