cgnorthcutt/benchmarking-keras-pytorch

🔥 Reproducibly benchmarking Keras and PyTorch models

benchmarking-keras-pytorch's Issues

Plans for including FLOps and Images/Second?

Hi. Thanks for this amazing resource.

Do you have plans to support measuring flops and images/second measurement?

why resnet architectures perform better in PyTorch and inception architectures perform better in Keras?

Image scaling for Inception model in PyTorch incorrect

For 299 size image you still resize the image to 256 before cropping to 299. You should resize to img_size / 0.875 before cropping. Some of the inception models can get a little better pushing closer to 0.9 for the ratio.

https://github.com/cgnorthcutt/benchmarking-keras-pytorch/blob/master/imagenet_pytorch_get_predictions.py#L95

In the past I've benchmarked Inception v3 weights in PyTorch at prec@1 77.436 prec@5 93.476

PyTorch ResNet50 Validation Accuracy

Hey there!

I came across your project from Jeremy Howard's Twitter. I think it's great to be benchmarking these numbers and keeping them in a single place!

I've tried running your script and ran into some problems that I was hoping you could help diagnose:
I ran python imagenet_pytorch_get_predictions.py -m resnet50 -g 0 -b 64 ~/imagenet/ and got

resnet50 completed: 100.00%
resnet50: acc1: 0.10%, acc5: 0.27%

I'm using Python 3.7 and PyTorch 1.0.1.post2 and didn't change any of your code except for making the argparse parameter for batch_size to be type=int.

I work pretty regularly with PyTorch and ResNet-50 and was surprised to see the ResNet-50 have only 75.02% validation accuracy. When I use the pretrained ResNet-50 using the code here, I get 76.138% top-1, 92.864% top-5 accuracy. Specifically, I run:

python main.py -a resnet50 -e -b 64 -j 8 --pretrained ~/imagenet/

I'm using CUDA 9.2 and CUDNN version 7.4.1 and running inference on a NVIDIA V100 on a Google Cloud instance using Ubuntu 16.04.

I'm curious what might be going wrong here and why our results are different - to start with, what version of CUDNN/CUDA did your results originate from?