Awesome - Most Cited Deep Learning Papers

A curated list of the most cited deep learning papers (since 2012)

We believe that there exist classic deep learning papers which are worth reading regardless of their application domain. Rather than providing overwhelming amount of papers, We would like to provide a curated list of the awewome deep learning papers (less than 150 papers) which are considered as must-reads in a certain researh domain.

Awesome list criteria

< 6 months : by discussion (See New Papers)
2016 : +30 citations (:sparkles: +60)
2015 : +100 citations (:sparkles: +200)
2014 : +200 citations (:sparkles: +400)
2013 : +300 citations (:sparkles: +600)
2012 : +400 citations (:sparkles: +800)
Before 2012 : Please refer to Classic papers section

Since the number of citations can be affected by the research domain, some important papers have been included in the list though they do not meet the criteria.

We need your contributions! If you find any missing papers, project links, names of distinguished researhers, etc., please feel free to edit and pull a request. (Please read the contributing guide for futher instructions, though letting me know only the title of papers is also a big contribution to us.)

Understanding / Generalization
Optimization / Training Technique
Unsupervised Learning / GAN
Image: Network Models
Image: Segmentation
Image: Object Detection
Image: Visual QnA, Caption
Video / Human Activity
Machine Translation / Word Embedding
Speech /
Reinforcement Learning / Robotics
Hardware / Software
New papers
Book / Survey / Review / Tutorial
Video Lecture / Tutorial
Classic Papers
Distinguished Researchers

Total 85 papers except for the papers in Hardware / Software, Papers Worth Reading, and Classic Papers sections.

New papers

Newly released papers (< 6 months) which are worth reading

Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models, S. Ioffe. (Google) [pdf]
Understanding deep learning requires rethinking generalization, C. Zhang et al. (Vinyals) [pdf]
WaveNet: A Generative Model for Raw Audio (2016), A. Oord et al. (DeepMind) [pdf] [web]
Layer Normalization (2016), J. Ba et al. (Hinton) [pdf]
Deep neural network architectures for deep reinforcement learning, Z. Wang et al. (DeepMind) [pdf]
Learning to learn by gradient descent by gradient descent (2016), M. Andrychowicz et al. (DeepMind) [pdf]
Adversarially learned inference (2016), V. Dumoulin et al. [web][pdf]
Understanding convolutional neural networks (2016), J. Koushik [pdf]
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 1MB model size (2016), F. Iandola et al. [pdf]
Learning to compose neural networks for question answering (2016), J. Andreas et al. [pdf]
Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection (2016) (Google), S. Levine et al. [pdf]
Taking the human out of the loop: A review of bayesian optimization (2016), B. Shahriari et al. [pdf]
Eie: Efficient inference engine on compressed deep neural network (2016), S. Han et al. [pdf]
Adaptive Computation Time for Recurrent Neural Networks (2016), A. Graves [pdf]
Densely connected convolutional networks (2016), G. Huang et al. [pdf]

Understanding / Generalization

Distilling the knowledge in a neural network (2015), G. Hinton et al. (Hinton, Vinyals, Dean: Google) [pdf] ✨
Deep neural networks are easily fooled: High confidence predictions for unrecognizable images (2015), A. Nguyen et al. [pdf]
How transferable are features in deep neural networks? (2014), J. Yosinski et al. (Bengio) [pdf]
Return of the devil in the details: delving deep into convolutional nets (2014), K. Chatfield et al. [pdf] ✨
Why does unsupervised pre-training help deep learning (2010), D. Erhan et al. (Bengio) [pdf]
Understanding the difficulty of training deep feedforward neural networks (2010), X. Glorot and Y. Bengio [pdf]

Optimization / Training Technique

Batch normalization: Accelerating deep network training by reducing internal covariate shift (2015), S. Loffe and C. Szegedy (Google) [pdf] ✨
Delving deep into rectifiers: Surpassing human-level performance on imagenet classification (2015), K. He et al. (He) [pdf] ✨
Recurrent neural network regularization (2014), W. Zaremba et al. (Sutskever, Vinyals: Google) [pdf]
Dropout: A simple way to prevent neural networks from overfitting (2014), N. Srivastava et al. (Hinton) [pdf] ✨
Adam: A method for stochastic optimization (2014), D. Kingma and J. Ba [pdf]
Spatial pyramid pooling in deep convolutional networks for visual recognition (2014), K. He et al. [pdf] ✨
On the importance of initialization and momentum in deep learning (2013), I. Sutskever et al. (Hinton) [pdf]
Regularization of neural networks using dropconnect (2013), L. Wan et al. (LeCun) [pdf]
Improving neural networks by preventing co-adaptation of feature detectors (2012), G. Hinton et al. [pdf] ✨
Random search for hyper-parameter optimization (2012) J. Bergstra and Y. Bengio [pdf]

Network Models

Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1 (2016), M. Courbariaux et al. [pdf]
Inception-v4, inception-resnet and the impact of residual connections on learning (2016), C. Szegedy et al. (Google) [pdf]
Identity Mappings in Deep Residual Networks (2016), K. He et al. (He) [pdf]
Deep residual learning for image recognition (2016), K. He et al. (He) [pdf] ✨
Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding (2015), S. Han et al. [pdf]
Going deeper with convolutions (2015), C. Szegedy et al. (Google) [pdf] ✨
An Empirical Exploration of Recurrent Network Architectures (2015), R. Jozefowicz et al. Sutskever: Google [pdf]
Fully convolutional networks for semantic segmentation (2015), J. Long et al. [pdf] ✨
Very deep convolutional networks for large-scale image recognition (2014), K. Simonyan and A. Zisserman [pdf] ✨
OverFeat: Integrated recognition, localization and detection using convolutional networks (2014), P. Sermanet et al. (LeCun) [pdf]
Visualizing and understanding convolutional networks (2014), M. Zeiler and R. Fergus [pdf] ✨
Maxout networks (2013), I. Goodfellow et al. (Bengio) [pdf]
Network in network (2013), M. Lin et al. [pdf]
ImageNet classification with deep convolutional neural networks (2012), A. Krizhevsky et al. (Hinton) [pdf] ✨
Large scale distributed deep networks (2012), J. Dean et al. [pdf] ✨
Deep sparse rectifier neural networks (2011), X. Glorot et al. (Bengio) [pdf]

Object Detection

Instance-aware semantic segmentation via multi-task network cascades (2016), J. Dai et al. [pdf]
Efficient piecewise training of deep structured models for semantic segmentation (2016), G. Lin et al. [pdf]
SSD: Single shot multibox detector (2016), W. Liu et al. [pdf]
You only look once: Unified, real-time object detection (2016), J. Redmon et al. [pdf]
Region-based convolutional networks for accurate object detection and segmentation (2016), R. Girshick et al. (He) [pdf]
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks (2015), S. Ren et al. [[pdf]] (http://papers.nips.cc/paper/5638-faster-r-cnn-towards-real-time-object-detection-with-region-proposal-networks.pdf) ✨
Fast R-CNN (2015), R. Girshick (He) [pdf] ✨
Scalable object detection using deep neural networks (2014), D. Erhan et al. (Google) [pdf]
Rich feature hierarchies for accurate object detection and semantic segmentation (2014), R. Girshick et al. [pdf] ✨
Semantic image segmentation with deep convolutional nets and fully connected CRFs, L. Chen et al. [pdf]
Deep neural networks for object detection (2013), C. Szegedy et al. [pdf]

Unsupervised / Adversarial

Generative visual manipulation on the natural image manifold (2016), J. Zhu et al. [pdf]
Conditional image generation with pixelcnn decoders (2016), A. van den Oord et al. [pdf]
Pixel recurrent neural networks (2016), A. van den Oord et al. (DeepMind) [pdf]
Unsupervised representation learning with deep convolutional generative adversarial networks (2015), A. Radford et al. [pdf]
DRAW: A recurrent neural network for image generation (2015), K. Gregor et al. [pdf]
CNN features off-the-Shelf: An astounding baseline for recognition (2014), A. Razavian et al. [pdf] ✨
Generative adversarial nets (2014), I. Goodfellow et al. (Bengio) [pdf]
Intriguing properties of neural networks (2014), C. Szegedy et al. (Sutskever, Goodfellow: Google) [pdf]
Auto-encoding variational Bayes (2013), D. Kingma and M. Welling [pdf]
Building high-level features usi ng large scale unsupervised learning (2013), Q. Le et al. [pdf] ✨
An analysis of single-layer networks in unsupervised feature learning (2011), A. Coates et al. [pdf]
Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion (2010), P. Vincent et al. (Bengio) [pdf]
A practical guide to training restricted boltzmann machines (2010), G. Hinton [pdf]

Image

Brain tumor segmentation with deep neural networks (2017), M. Havaei et al. (Bengio) [pdf]
Weakly supervised object localization with multi-fold multiple instance learning (2017), R. Gokberk et al. [pdf]
Inside-outside net: Detecting objects in context with skip pooling and recurrent neural networks (2016), S. Bell et al. [pdf].
Perceptual losses for real-time style transfer and super-resolution (2016), J. Johnson et al. [pdf]
Colorful image colorization (2016), R. Zhang et al. [pdf]
What makes for effective detection proposals? (2016), J. Hosan et al. (Facebook) [pdf]
Image Super-Resolution Using Deep Convolutional Networks (2016), C. Dong et al. (He) [pdf] ✨
Reading text in the wild with convolutional neural networks (2016), M. Jaderberg et al. (DeepMind) [pdf]
A neural algorithm of artistic style (2015), L. Gatys et al. [pdf]
Learning Deconvolution Network for Semantic Segmentation (2015), H. Noh et al. [pdf]
Imagenet large scale visual recognition challenge (2015), O. Russakovsky et al. [pdf] ✨
Learning a Deep Convolutional Network for Image Super-Resolution (2014, C. Dong et al. (He) [pdf]
Learning a Deep Convolutional Network for Image Super-Resolution (2014), C. Dong et al. [pdf]
Learning and transferring mid-Level image representations using convolutional neural networks (2014), M. Oquab et al. [pdf]
DeepFace: Closing the Gap to Human-Level Performance in Face Verification (2014), Y. Taigman et al. (Facebook) [pdf] ✨
Decaf: A deep convolutional activation feature for generic visual recognition (2013), J. Donahue et al. [pdf] ✨
Learning hierarchical features for scene labeling (2013), C. Farabet et al. (LeCun) [pdf]
Learning mid-level features for recognition (2010), Y. Boureau (LeCun) [pdf]

Caption / Visual QnA

Mind's eye: A recurrent visual representation for image caption generation (2015), X. Chen and C. Zitnick. [pdf]
From captions to visual concepts and back (2015), H. Fang et al. [pdf].
VQA: Visual question answering (2015), S. Antol et al. [pdf]
Towards ai-complete question answering: A set of prerequisite toy tasks (2015), J. Weston et al. (Mikolov: Facebook) [pdf]
Ask me anything: Dynamic memory networks for natural language processing (2015), A. Kumar et al. [pdf]
A large annotated corpus for learning natural language inference (2015), S. Bowman et al. [pdf]
Show, attend and tell: Neural image caption generation with visual attention (2015), K. Xu et al. (Bengio) [pdf] ✨
Show and tell: A neural image caption generator (2015), O. Vinyals et al. (Vinyals: Google) [pdf] ✨
Long-term recurrent convolutional networks for visual recognition and description (2015), J. Donahue et al. [pdf] ✨
Deep visual-semantic alignments for generating image descriptions (2015), A. Karpathy and L. Fei-Fei [pdf] ✨
Deep captioning with multimodal recurrent neural networks (m-rnn) (2014), J. Mao et al. [pdf]

Video / Human Activity

Beyond short snippents: Deep networks for video classification (2015) (Vinyals: Google) [pdf] ✨
Large-scale video classification with convolutional neural networks (2014), A. Karpathy et al. (FeiFei) [pdf] ✨
DeepPose: Human pose estimation via deep neural networks (2014), A. Toshev and C. Szegedy (Google) [pdf]
Two-stream convolutional networks for action recognition in videos (2014), K. Simonyan et al. [pdf]
A survey on human activity recognition using wearable sensors (2013), O. Lara and M. Labrador [pdf]
3D convolutional neural networks for human action recognition (2013), S. Ji et al. [pdf]
Action recognition with improved trajectories (2013), H. Wang and C. Schmid [pdf]
Learning hierarchical invariant spatio-temporal features for action recognition with independent subspace analysis (2011), Q. Le et al. [pdf]

Word Embedding

Glove: Global vectors for word representation (2014), J. Pennington et al. [pdf] ✨
Distributed representations of sentences and documents (2014), Q. Le and T. Mikolov (Le, Mikolov: Google) [pdf] (Google) ✨
Distributed representations of words and phrases and their compositionality (2013), T. Mikolov et al. (Google) [pdf] ✨
Efficient estimation of word representations in vector space (2013), T. Mikolov et al. (Google) [pdf] ✨
Devise: A deep visual-semantic embedding model (2013), A. Frome et al., (Mikolov: Google) [pdf]
Word representations: a simple and general method for semi-supervised learning (2010), J. Turian (Bengio) [pdf]

Machine Translation / QnA

A Character-Level Decoder without Explicit Segmentation for Neural Machine Translation (2016), J. Chung et al. [[pdf]]
A thorough examination of the cnn/daily mail reading comprehension task (2016), D. Chen et al. [pdf]
Achieving open vocabulary neural machine translation with hybrid word-character models, M. Luong and C. Manning. [pdf]
Very Deep Convolutional Networks for Natural Language Processing (2016), A. Conneau et al. [pdf]
Bag of tricks for efficient text classification (2016), A. Joulin et al. [pdf]
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016), Y. Wu et al. (Le, Vinyals, Dean: Google) [pdf]
Exploring the limits of language modeling (2016), R. Jozefowicz et al. (Vinyals: DeepMind) [pdf]
Teaching machines to read and comprehend, K. Hermann et al. [pdf]
Effective approaches to attention-based neural machine translation (2015), M. Luong et al. [pdf]
A neural conversational model (2015), O. Vinyals and Q. Le. (Vinyals, Le: Google) [pdf]
Character-aware neural language models (2015), Y. Kim et al. [pdf]
Grammar as a foreign language (2015), O. Vinyals et al. (Vinyals, Sutskever, Hinton: Google) [pdf]
Towards AI-complete question answering: A set of prerequisite toy tasks (2015), J. Weston et al. [pdf]
Neural machine translation by jointly learning to align and translate (2014), D. Bahdanau et al. (Bengio) [pdf] ✨
Sequence to sequence learning with neural networks (2014), I. Sutskever et al. (Sutskever, Vinyals, Le: Google) [pdf] ✨
Learning phrase representations using RNN encoder-decoder for statistical machine translation (2014), K. Cho et al. (Bengio) [pdf]
A convolutional neural network for modelling sentences (2014), N. Kalchbrenner et al. [pdf]
Convolutional neural networks for sentence classification (2014), Y. Kim [pdf]
Addressing the rare word problem in neural machine translation (2014), M. Luong et al. [pdf]
The stanford coreNLP natural language processing toolkit (2014), C. Manning et al. [pdf] ✨
Recursive deep models for semantic compositionality over a sentiment treebank (2013), R. Socher et al. [pdf] ✨
Linguistic Regularities in Continuous Space Word Representations (2013), T. Mikolov et al. (Mikolov: Microsoft) [pdf]
Natural language processing (almost) from scratch (2011), R. Collobert et al. [pdf] ✨
Recurrent neural network based language model (2010), T. Mikolov et al. [pdf]

RNN / LSTM / Memory Network

Hybrid computing using a neural network with dynamic external memory (2016), A. Graves et al. [pdf]
Improved semantic representations from tree-structured long short-term memory networks (2015), K. Tai et al. [pdf]
End-to-end memory networks (2015), S. Sukbaatar et al. [pdf]
Conditional random fields as recurrent neural networks (2015), S. Zheng and S. Jayasumana. [pdf] ✨
Memory networks (2014), J. Weston et al. [pdf]
Neural turing machines (2014), A. Graves et al. [pdf]

Speech / Etc.

Automatic speech recognition - A deep learning approach (Book, 2015), D. Yu and L. Deng. [html]
Deep speech 2: End-to-end speech recognition in English and Mandarin (2015), D. Amodei et al. [pdf]
Towards end-to-end speech recognition with recurrent neural networks (2014), A. Graves and N. Jaitly. [pdf]
Speech recognition with deep recurrent neural networks (2013), A. Graves (Hinton) [pdf]
Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups (2012), G. Hinton et al. [pdf] ✨
Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition (2012) G. Dahl et al. [pdf] ✨
Acoustic modeling using deep belief networks (2012), A. Mohamed et al. (Hinton) [pdf]

RL / Robotics

End-to-end training of deep visuomotor policies (2016), S. Levine et al. [pdf] ✨
Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection (2016), S. Levine et al. [pdf]
Continuous deep q-learning with model-based acceleration (2016), S. Gu et al. [pdf]
Continuous deep q-learning with model-based acceleration (2016), S. Gu et al. [pdf]
Asynchronous methods for deep reinforcement learning (2016), V. Mnih et al. [pdf]
Mastering the game of Go with deep neural networks and tree search (2016), D. Silver et al. (Sutskever: DeepMind) [pdf] ✨
Continuous control with deep reinforcement learning (2015), T. Lillicrap et al. [pdf]
Trust Region Policy Optimization (2015), J. Schulman et al. [pdf]
Human-level control through deep reinforcement learning (2015), V. Mnih et al. (DeepMind) [pdf] ✨
Deep learning for detecting robotic grasps (2015), I. Lenz et al. [pdf]
Playing atari with deep reinforcement learning (2013), V. Mnih et al. [pdf])

Hardware / Software

OpenAI gym (2016), G. Brockman et al. [pdf]
TensorFlow: Large-scale machine learning on heterogeneous distributed systems (2016), M. Abadi et al. [pdf] ✨
Theano: A Python framework for fast computation of mathematical expressions, R. Al-Rfou et al. (Bengio)
MatConvNet: Convolutional neural networks for matlab (2015), A. Vedaldi and K. Lenc [pdf]
Caffe: Convolutional architecture for fast feature embedding (2014), Y. Jia et al. [pdf] ✨

Book / Survey / Review / Tutorial / Blog

Deep learning (Book, 2016), Goodfellow et al. [html]
Deep learning (2015), Y. LeCun, Y. Bengio and G. Hinton [pdf] ✨
Deep learning in neural networks: An overview (2015), J. Schmidhuber [pdf] ✨
Representation learning: A review and new perspectives (2013), Y. Bengio et al. [pdf] ✨

Classic Papers

Classic papers (1997~2011) which cause the advent of deep learning era

Recurrent neural network based language model (2010), T. Mikolov et al. [pdf]
Learning deep architectures for AI (2009), Y. Bengio. [pdf]
Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations (2009), H. Lee et al. [pdf]
Greedy layer-wise training of deep networks (2007), Y. Bengio et al. [pdf]
Reducing the dimensionality of data with neural networks, G. Hinton and R. Salakhutdinov. [pdf]
A fast learning algorithm for deep belief nets (2006), G. Hinton et al. [pdf]
Gradient-based learning applied to document recognition (1998), Y. LeCun et al. [pdf]
Long short-term memory (1997), S. Hochreiter and J. Schmidhuber. [pdf]

Distinguished Researchers

Distinguished deep learning researchers who have published +3 (:sparkles: +6) papers on the awesome list (The papers in Hardware / Software, Papers Worth Reading, Classic Papers sections are excluded in counting.)

Chirstian Szegedy, Google ✨
Kaiming He, Facebook ✨
Geoffrey Hinton, Google ✨
Ilya Sutskever, OpenAI ✨
Ian Goodfellow, OpenAI ✨
Oriol Vinyals, Google DeepMind ✨
Quoc Le, Google ✨
Tomas Mikolov, Facebook
Yann LeCun, Facebook ✨
Yoshua Bengio, University of Montreal ✨
Aaron Courville, University of Montreal
Alex Graves, Google DeepMind
Andrej Karpathy, OpenAI
Andrew Ng, Baidu
Andrew Zisserman, University of Oxford
Christopher Manning, Stanford University
David Silver, Google DeepMind
Dong Yu, Microsoft Research
Ross Girshick, Facebook
Karen Simonyan, Google DeepMind
Kyunghyun Cho, New York University
Honglak Lee, University of Michigan
Jeff Dean, Google,
Jeff Donahue, U.C. Berkeley
Jian Sun, Microsoft Research
Juergen Schmidhuber, Swiss AI Lab IDSIA
Li Fei-Fei, Stanford University
Pascal Vincent, University of Montreal
Rob Fergus, Facebook, New York University
Ruslan Salakhutdinov, CMU
Trevor Darrell, U.C. Berkeley

Acknowledgement

Thank you for all your contributions. Please make sure to read the contributing guide before you make a pull request.

You can follow my facebook page, twitter or google plus to get useful information about machine learning and deep learning.

License

To the extent possible under law, Terry T. Um has waived all copyright and related or neighboring rights to this work.

ercanxcan / awesome-deep-learning-papers Goto Github PK

awesome-deep-learning-papers's Introduction