ibrahimzamit/awesome-autodl

A curated list of automated deep learning (including neural architecture search and hyper-parameter optimization) resources.

Awesome AutoDL

A curated list of automated deep learning related resources. Inspired by awesome-deep-vision, awesome-adversarial-machine-learning, awesome-deep-learning-papers, and awesome-architecture-search.

Please feel free to pull requests or open an issue to add papers.

Table of Contents

• Awesome Blogs

• Awesome AutoDL Libraies

• Deep Learning-based NAS and HPO

  • 2020 Venues
  • 2019 Venues
  • 2018 Venues
  • 2017 Venues
  • Previous Venues
  • arXiv

• Awesome Surveys

Awesome Blogs

• AutoML info and AutoML Freiburg-Hannover
• What’s the deal with Neural Architecture Search?
• Google Could AutoML and PocketFlow
• AutoML Challenge and AutoDL Challenge

Awesome AutoDL Libraies

• Keras Tuner
• NNI
• AutoGluon
• Auto-PyTorch
• AutoDL-Projects

Deep Learning-based NAS and HPO

Type	G	RL	EA	PD	Other
Explanation	gradient-based	reinforcement learning	evolutionary algorithm	performance prediction	other types

2020

Title	Venue	Type	Code
Optimizing Millions of Hyperparameters by Implicit Differentiation	AISTATS	G	-
Evolving Machine Learning Algorithms From Scratch	ICML	-	-
NADS: Neural Architecture Distribution Search for Uncertainty Awareness	ICML	-	-
Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data	ICML	-	-
Neural Architecture Search in a Proxy Validation Loss Landscape	ICML	-	-
MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation	CVPR	G	GitHub
Binarizing MobileNet via Evolution-based Searching	CVPR	EA	-
Rethinking Performance Estimation in Neural Architecture Search	CVPR	-	GitHub
APQ: Joint Search for Network Architecture, Pruning and Quantization Policy	CVPR	G	GitHub
SGAS: Sequential Greedy Architecture Search	CVPR	G	Github
Can Weight Sharing Outperform Random Architecture Search? An Investigation With TuNAS	CVPR	RL	-
FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions	CVPR	G	Github
AdversarialNAS: Adversarial Neural Architecture Search for GANs	CVPR	G	Github
When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks	CVPR	G	Github
Block-wisely Supervised Neural Architecture Search with Knowledge Distillation	CVPR	G	Github
Overcoming Multi-Model Forgetting in One-Shot NAS with Diversity Maximization	CVPR	G	Github
Densely Connected Search Space for More Flexible Neural Architecture Search	CVPR	G	Github
EfficientDet: Scalable and Efficient Object Detection	CVPR	RL	-
NAS-BENCH-201: Extending the Scope of Reproducible Neural Architecture Search	ICLR	-	Github
Understanding Architectures Learnt by Cell-based Neural Architecture Search	ICLR	G	GitHub
Evaluating The Search Phase of Neural Architecture Search	ICLR	-
AtomNAS: Fine-Grained End-to-End Neural Architecture Search	ICLR		GitHub
Fast Neural Network Adaptation via Parameter Remapping and Architecture Search	ICLR	-	GitHub
Once for All: Train One Network and Specialize it for Efficient Deployment	ICLR	G	GitHub
Efficient Transformer for Mobile Applications	ICLR	-	-
PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search	ICLR	G	GitHub
Adversarial AutoAugment	ICLR	-	-
NAS evaluation is frustratingly hard	ICLR	-	GitHub
FasterSeg: Searching for Faster Real-time Semantic Segmentation	ICLR	G	GitHub
Computation Reallocation for Object Detection	ICLR	-	-
Towards Fast Adaptation of Neural Architectures with Meta Learning	ICLR	-	-
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures	ICLR	EA	-
How to Own the NAS in Your Spare Time	ICLR	-	-

2019

Title	Venue	Type	Code
Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions	ICLR	-	-
DATA: Differentiable ArchiTecture Approximation	NeurIPS	-	-
Random Search and Reproducibility for Neural Architecture Search	UAI	G	GitHub
Improved Differentiable Architecture Search for Language Modeling and Named Entity Recognition	EMNLP	G	-
Continual and Multi-Task Architecture Search	ACL	RL	-
Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation	ICCV	-	-
Multinomial Distribution Learning for Effective Neural Architecture Search	ICCV	-	-
Searching for MobileNetV3	ICCV	EA	-
Multinomial Distribution Learning for Effective Neural Architecture Search	ICCV	-	GitHub
Fast and Practical Neural Architecture Search	ICCV
Teacher Guided Architecture Search	ICCV		-
AutoDispNet: Improving Disparity Estimation With AutoML	ICCV	G	-
Resource Constrained Neural Network Architecture Search: Will a Submodularity Assumption Help?	ICCV	EA	-
One-Shot Neural Architecture Search via Self-Evaluated Template Network	ICCV	G	Github
Evolving Space-Time Neural Architectures for Videos	ICCV	EA	GitHub
AutoGAN: Neural Architecture Search for Generative Adversarial Networks	ICCV	RL	github
Discovering Neural Wirings	NeurIPS	G	Github
Towards modular and programmable architecture search	NeurIPS	Other	Github
Network Pruning via Transformable Architecture Search	NeurIPS	G	Github
Deep Active Learning with a NeuralArchitecture Search	NeurIPS	-	-
DetNAS: Backbone Search for ObjectDetection	NeurIPS	-	-
SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers	NeurIPS	-	-
Efficient Forward Architecture Search	NeurIPS	G	Github
Efficient Neural ArchitectureTransformation Search in Channel-Level for Object Detection	NeurIPS	G	-
XNAS: Neural Architecture Search with Expert Advice	NeurIPS	G	-
DARTS: Differentiable Architecture Search	ICLR	G	github
ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware	ICLR	RL/G	github
Graph HyperNetworks for Neural Architecture Search	ICLR	G	-
Learnable Embedding Space for Efficient Neural Architecture Compression	ICLR	Other	github
Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution	ICLR	EA	-
SNAS: stochastic neural architecture search	ICLR	G	-
NetTailor: Tuning the Architecture, Not Just the Weights	CVPR	G	Github
Searching for A Robust Neural Architecture in Four GPU Hours	CVPR	G	Github
ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation	CVPR	-	-
Partial Order Pruning: for Best Speed/Accuracy Trade-off in Neural Architecture Search	CVPR	EA	github
FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search	CVPR	G	-
RENAS: Reinforced Evolutionary Neural Architecture Search	CVPR	G	-
Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation	CVPR	G	GitHub
MnasNet: Platform-Aware Neural Architecture Search for Mobile	CVPR	RL	Github
MFAS: Multimodal Fusion Architecture Search	CVPR	EA	-
A Neurobiological Evaluation Metric for Neural Network Model Search	CVPR	Other	-
Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cells	CVPR	RL	-
Customizable Architecture Search for Semantic Segmentation	CVPR	-	-
Regularized Evolution for Image Classifier Architecture Search	AAAI	EA	-
AutoAugment: Learning Augmentation Policies from Data	CVPR	RL	-
Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules	ICML	EA	-
The Evolved Transformer	ICML	EA	Github
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks	ICML	RL	-
NAS-Bench-101: Towards Reproducible Neural Architecture Search	ICML	Other	Github

2018

Title	Venue	Type	Code
Towards Automatically-Tuned Deep Neural Networks	BOOK	-	GitHub
Efficient Architecture Search by Network Transformation	AAAI	RL	github
Learning Transferable Architectures for Scalable Image Recognition	CVPR	RL	github
N2N learning: Network to Network Compression via Policy Gradient Reinforcement Learning	ICLR	RL	-
A Flexible Approach to Automated RNN Architecture Generation	ICLR	RL/PD	-
Practical Block-wise Neural Network Architecture Generation	CVPR	RL	-
Path-Level Network Transformation for Efficient Architecture Search	ICML	RL	github
Hierarchical Representations for Efficient Architecture Search	ICLR	EA	-
Understanding and Simplifying One-Shot Architecture Search	ICML	G	-
SMASH: One-Shot Model Architecture Search through HyperNetworks	ICLR	G	github
Neural Architecture Optimization	NeurIPS	G	github
Searching for efficient multi-scale architectures for dense image prediction	NeurIPS	Other	-
Progressive Neural Architecture Search	ECCV	PD	github
Neural Architecture Search with Bayesian Optimisation and Optimal Transport	NeurIPS	Other	github
Differentiable Neural Network Architecture Search	ICLR-W	G	-
Accelerating Neural Architecture Search using Performance Prediction	ICLR-W	PD	-

2017

Title	Venue	Type	Code
Neural Architecture Search with Reinforcement Learning	ICLR	RL	-
Designing Neural Network Architectures using Reinforcement Learning	ICLR	RL	-
Neural Optimizer Search with Reinforcement Learning	ICML	RL	-
Learning Curve Prediction with Bayesian Neural Networks	ICLR	PD	-
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization	ICLR	PD	-
Hyperparameter Optimization: A Spectral Approach	NeurIPS-W	Other	github
Learning to Compose Domain-Specific Transformations for Data Augmentation	NeurIPS	-	-

2012-2016

Title	Venue	Type	Code
Speeding up Automatic Hyperparameter Optimization of Deep Neural Networksby Extrapolation of Learning Curves	IJCAI	PD	github

arXiv

Title	Date	Type	Code
AutoHAS: Differentiable Hyper-parameter and Architecture Search	2020.06	G	-
FBNetV3: Joint Architecture-Recipe Search using Neural Acquisition Function	2020.06	PD	-
Population Based Training of Neural Networks	2017.11	EA	-
NSGA-NET: A Multi-Objective Genetic Algorithm for Neural Architecture Search	2018.10	EA	-
Training Frankenstein’s Creature to Stack: HyperTree Architecture Search	2018.10	G	-
Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search	2019.01	G	github

Awesome Surveys

Title	Venue	Year
Automated Machine Learning	Springer Book	2019
Neural architecture search: A survey	JMLR	2019
Taking human out of learning applications: A survey on automated machine learning	arXiv	2018
A Survey on Neural Architecture Search	arXiv	2019
A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions	arXiv	2020