Awesome AutoDL

A curated list of automated deep learning related resources. Inspired by awesome-deep-vision, awesome-adversarial-machine-learning, awesome-deep-learning-papers, and awesome-architecture-search.

Please feel free to pull requests or open an issue to add papers.

---

Table of Contents

• Awesome Blogs
• Awesome AutoDL Libraies
• Awesome Benchmarks
• Deep Learning-based NAS and HPO
• 2021 Venues
• 2020 Venues
• 2019 Venues
• 2018 Venues
• 2017 Venues
• Previous Venues
• arXiv
• Awesome Surveys

---

Awesome Blogs

• AutoML info and AutoML Freiburg-Hannover
• What’s the deal with Neural Architecture Search?
• Google Could AutoML and PocketFlow
• AutoML Challenge and AutoDL Challenge
• In Defense of Weight-sharing for Neural Architecture Search: an optimization perspective

Awesome AutoDL Libraies

• PyGlove
• NASLib
• Keras Tuner
• NNI
• AutoGluon
• Auto-PyTorch
• AutoDL-Projects
• aw_nas
• Determined
• TPOT

Awesome Benchmarks

| Title | Venue | Code | |:--------|:--------:|:--------:| | NAS-Bench-101: Towards Reproducible Neural Architecture Search | ICML 2019 | GitHub | | NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search | ICLR 2020 | Github | | NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search | arXiv 2020 | GitHub | | NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural Architecture Search | ICLR 2020 | GitHub | | NATS-Bench: Benchmarking NAS Algorithms for Architecture Topology and Size | TPAMI 2021 | GitHub | NAS-Bench-ASR: Reproducible Neural Architecture Search for Speech Recognition | ICLR 2021 | GitHub | | HW-NAS-Bench: Hardware-Aware Neural Architecture Search Benchmark | ICLR 2021 | GitHub | | NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing | arXiv 2020 | GitHub | | NAS-Bench-x11 and the Power of Learning Curves | NeurIPS 2021 | GitHub |

Deep Learning-based NAS and HPO

| Type | G | RL | EA | PD | Other | |:------------|:--------------:|:----------------------:|:-----------------------:|:----------------------:|:----------:| | Explanation | gradient-based | reinforcement learning | evolutionary algorithm | performance prediction | other types |

2021 Venues

| Title | Venue | Type | Code | |:--------|:--------:|:--------:|:--------:| | CATE: Computation-aware Neural Architecture Encoding with Transformers | ICML | O | GitHub | | Searching by Generating: Flexible and Efficient One-Shot NAS with Architecture Generator | CVPR | G | Github | | Zen-NAS: A Zero-Shot NAS for High-Performance Deep Image Recognition | ICCV | EA | Github | | AutoFormer: Searching Transformers for Visual Recognition |ICCV | EA | GitHub | LightTrack: Finding Lightweight Neural Networks for Object Tracking via One-Shot Architecture Search | CVPR | EA | GitHub | | One-Shot Neural Ensemble Architecture Search by Diversity-Guided Search Space Shrinking | CVPR | EA | GitHub | | DARTS-: Robustly Stepping out of Performance Collapse Without Indicators | ICLR | G | GitHub | | Zero-Cost Proxies for Lightweight NAS | ICLR | O | GitHub | | Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective | ICLR | - | GitHub | | DrNAS: Dirichlet Neural Architecture Search | ICLR | G | GitHub | | Rethinking Architecture Selection in Differentiable NAS | ICLR | O | GitHub | | Evolving Reinforcement Learning Algorithms | ICLR | EA | GitHub | | AutoHAS: Differentiable Hyper-parameter and Architecture Search | ICLR-W | G | - | | FBNetV3: Joint Architecture-Recipe Search using Neural Acquisition Function | CVPR | PD | github |

2020 Venues

| Title | Venue | Type | Code | |:--------|:--------:|:--------:|:--------:| | Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search | NeurIPS | - | GitHub | | PyGlove: Symbolic Programming for Automated Machine Learning | NeurIPS | library | - | | Does Unsupervised Architecture Representation Learning Help Neural Architecture Search | NeurIPS | PD | GitHub | | RandAugment: Practical Automated Data Augmentation with a Reduced Search Space | NeurIPS | | GitHub | | Delta-STN: Efficient Bilevel Optimization for Neural Networks using Structured Response Jacobians | NeurIPS | G | GitHub | | A Study on Encodings for Neural Architecture Search | NeurIPS | | GitHub | | AutoBSS: An Efficient Algorithm for Block Stacking Style Search | NeurIPS | | | | Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS | NeurIPS | G | GitHub | | Interstellar: Searching Recurrent Architecture for Knowledge Graph Embedding | NeurIPS | | | | Revisiting Parameter Sharing for Automatic Neural Channel Number Search | NeurIPS | | | | Learning Search Space Partition for Black-box Optimization using Monte Carlo Tree Search | NeurIPS | MCTS | GitHub | | Neural Architecture Search using Deep Neural Networks and Monte Carlo Tree Search | AAAI | MCTS | GitHub | | Representation Sharing for Fast Object Detector Search and Beyond | ECCV | G | GitHub | | Are Labels Necessary for Neural Architecture Search? | ECCV | G | - | | Single Path One-Shot Neural Architecture Search with Uniform Sampling | ECCV | EA | - | | Neural Predictor for Neural Architecture Search | ECCV | O | - | | BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models | ECCV | G | - | | BATS: Binary ArchitecTure Search | ECCV | - | - | | AttentionNAS: Spatiotemporal Attention Cell Search for Video Classification | ECCV | - | - | | Search What You Want: Barrier Panelty NAS for Mixed Precision Quantization | ECCV | - | - | | Angle-based Search Space Shrinking for Neural Architecture Search | ECCV | - | - | | Anti-Bandit Neural Architecture Search for Model Defense | ECCV | - | - | | TF-NAS: Rethinking Three Search Freedoms of Latency-Constrained Differentiable Neural Architecture Search | ECCV | G | GitHub | | Fair DARTS: Eliminating Unfair Advantages in Differentiable Architecture Search | ECCV | G | GitHub | | Off-Policy Reinforcement Learning for Efficient and Effective GAN Architecture Search | ECCV | RL | - | | DA-NAS: Data Adapted Pruning for Efficient Neural Architecture Search | ECCV | G | - | | Optimizing Millions of Hyperparameters by Implicit Differentiation | AISTATS | G | - | | Evolving Machine Learning Algorithms From Scratch | ICML | EA | - | | Stabilizing Differentiable Architecture Search via Perturbation-based Regularization | ICML | G | GitHub | | NADS: Neural Architecture Distribution Search for Uncertainty Awareness | ICML | - | - | | Generative Teaching Networks: Accelerating Neural Architecture Search by Learning to Generate Synthetic Training Data | ICML | - | - | | Neural Architecture Search in a Proxy Validation Loss Landscape | ICML | - | - | | Hit-Detector: Hierarchical Trinity Architecture Search for Object Detection | CVPR | - | GitHub | | Designing Network Design Spaces | CVPR | - | GitHub | | UNAS: Differentiable Architecture Search Meets Reinforcement Learning | CVPR | G/RL | GitHub | | MiLeNAS: Efficient Neural Architecture Search via Mixed-Level Reformulation | CVPR | G | GitHub | | A Semi-Supervised Assessor of Neural Architectures | CVPR | PD | - | | Binarizing MobileNet via Evolution-based Searching | CVPR | EA | - | | Rethinking Performance Estimation in Neural Architecture Search | CVPR | - | GitHub | | APQ: Joint Search for Network Architecture, Pruning and Quantization Policy | CVPR | G | GitHub | | SGAS: Sequential Greedy Architecture Search | CVPR | G | Github | | Can Weight Sharing Outperform Random Architecture Search? An Investigation With TuNAS | CVPR | RL | - | | FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions | CVPR | G | Github | | AdversarialNAS: Adversarial Neural Architecture Search for GANs | CVPR | G | Github | | When NAS Meets Robustness: In Search of Robust Architectures against Adversarial Attacks | CVPR | G | Github | | Block-wisely Supervised Neural Architecture Search with Knowledge Distillation | CVPR | G | Github | | Overcoming Multi-Model Forgetting in One-Shot NAS with Diversity Maximization | CVPR | G | Github | | Densely Connected Search Space for More Flexible Neural Architecture Search | CVPR | G | Github | | EfficientDet: Scalable and Efficient Object Detection | CVPR | RL | - | | NAS-BENCH-201: Extending the Scope of Reproducible Neural Architecture Search | ICLR | - | Github | | Understanding Architectures Learnt by Cell-based Neural Architecture Search | ICLR | G | GitHub | | Evaluating The Search Phase of Neural Architecture Search | ICLR | - | | | AtomNAS: Fine-Grained End-to-End Neural Architecture Search | ICLR | | GitHub | | Fast Neural Network Adaptation via Parameter Remapping and Architecture Search | ICLR | - | GitHub | | Once for All: Train One Network and Specialize it for Efficient Deployment | ICLR | G | GitHub | | Efficient Transformer for Mobile Applications | ICLR | - | - | | PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search | ICLR | G | GitHub | | Adversarial AutoAugment | ICLR | - | - | | NAS evaluation is frustratingly hard | ICLR | - | GitHub | | FasterSeg: Searching for Faster Real-time Semantic Segmentation | ICLR | G | GitHub | | Computation Reallocation for Object Detection | ICLR | - | - | | Towards Fast Adaptation of Neural Architectures with Meta Learning | ICLR | - | GitHub | | AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures | ICLR | EA | - | | How to Own the NAS in Your Spare Time | ICLR | - | - | | Fast, Accurate and Lightweight Super-Resolution with Neural Architecture Search | ICPR | G | github |

2019 Venues

| Title | Venue | Type | Code | |:--------|:--------:|:--------:|:--------:| | Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions | ICLR | - | - | | DATA: Differentiable ArchiTecture Approximation | NeurIPS | - | - | | Random Search and Reproducibility for Neural Architecture Search | UAI | G | GitHub | | Improved Differentiable Architecture Search for Language Modeling and Named Entity Recognition | EMNLP | G | - | | Continual and Multi-Task Architecture Search | ACL | RL | - | | Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation | ICCV | G | GitHub | | Multinomial Distribution Learning for Effective Neural Architecture Search | ICCV | - | GitHub | | Searching for MobileNetV3 | ICCV | EA | - | | Multinomial Distribution Learning for Effective Neural Architecture Search | ICCV | - | GitHub | | Fast and Practical Neural Architecture Search | ICCV | | | | Teacher Guided Architecture Search | ICCV | | - | | AutoDispNet: Improving Disparity Estimation With AutoML | ICCV | G | - | | Resource Constrained Neural Network Architecture Search: Will a Submodularity Assumption Help? | ICCV | EA | - | | One-Shot Neural Architecture Search via Self-Evaluated Template Network | ICCV | G | Github | | Evolving Space-Time Neural Architectures for Videos | ICCV | EA | GitHub | | AutoGAN: Neural Architecture Search for Generative Adversarial Networks | ICCV | RL | github | | Discovering Neural Wirings | NeurIPS | G | Github | | Towards modular and programmable architecture search | NeurIPS | Other | Github | | Network Pruning via Transformable Architecture Search | NeurIPS | G | Github | | Deep Active Learning with a NeuralArchitecture Search | NeurIPS | - | - | | DetNAS: Backbone Search for Object Detection | NeurIPS | EA | GitHub | | SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers | NeurIPS | - | - | | Efficient Forward Architecture Search | NeurIPS | G | Github | | Efficient Neural ArchitectureTransformation Search in Channel-Level for Object Detection | NeurIPS | G | - | | XNAS: Neural Architecture Search with Expert Advice | NeurIPS | G | GitHub | | DARTS: Differentiable Architecture Search | ICLR | G | github | | ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware | ICLR | RL/G | github | | Graph HyperNetworks for Neural Architecture Search | ICLR | G | - | | Learnable Embedding Space for Efficient Neural Architecture Compression | ICLR | Other | github | | Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution | ICLR | EA | - | | SNAS: stochastic neural architecture search | ICLR | G | - | | NetTailor: Tuning the Architecture, Not Just the Weights | CVPR | G | Github | | Searching for A Robust Neural Architecture in Four GPU Hours | CVPR | G | Github | | ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation | CVPR | - | - | | Partial Order Pruning: for Best Speed/Accuracy Trade-off in Neural Architecture Search | CVPR | EA | github | | FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search | CVPR | G | - | | RENAS: Reinforced Evolutionary Neural Architecture Search | CVPR | G | - | | Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation | CVPR | G | GitHub | | MnasNet: Platform-Aware Neural Architecture Search for Mobile | CVPR | RL | Github | | MFAS: Multimodal Fusion Architecture Search | CVPR | EA | - | | A Neurobiological Evaluation Metric for Neural Network Model Search | CVPR | Other | - | | Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cells | CVPR | RL | - | | Customizable Architecture Search for Semantic Segmentation | CVPR | - | - | | Regularized Evolution for Image Classifier Architecture Search | AAAI | EA | - | | AutoAugment: Learning Augmentation Policies from Data | CVPR | RL | - | | Population Based Augmentation: Efficient Learning of Augmentation Policy Schedules | ICML | EA | - | | The Evolved Transformer | ICML | EA | Github | | EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks | ICML | RL | - | | NAS-Bench-101: Towards Reproducible Neural Architecture Search | ICML | Other | Github | | On Network Design Spaces for Visual Recognition | ICCV | G | Github |

2018 Venues

| Title | Venue | Type | Code | |:--------|:--------:|:--------:|:--------:| | Towards Automatically-Tuned Deep Neural Networks | BOOK | - | GitHub | | NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications | ECCV | - | github | | Efficient Architecture Search by Network Transformation | AAAI | RL | github | | Learning Transferable Architectures for Scalable Image Recognition | CVPR | RL | github | | N2N learning: Network to Network Compression via Policy Gradient Reinforcement Learning | ICLR | RL | - | | A Flexible Approach to Automated RNN Architecture Generation | ICLR | RL/PD | - | | Practical Block-wise Neural Network Architecture Generation | CVPR | RL | - | Efficient Neural Architecture Search via Parameter Sharing | ICML | RL | github | | Path-Level Network Transformation for Efficient Architecture Search | ICML | RL | github | | Hierarchical Representations for Efficient Architecture Search | ICLR | EA | - | | Understanding and Simplifying One-Shot Architecture Search | ICML | G | - | | SMASH: One-Shot Model Architecture Search through HyperNetworks | ICLR | G | github | | Neural Architecture Optimization | NeurIPS | G | github | | Searching for efficient multi-scale architectures for dense image prediction | NeurIPS | Other | - | | Progressive Neural Architecture Search | ECCV | PD | github | | Neural Architecture Search with Bayesian Optimisation and Optimal Transport | NeurIPS | Other | github | | Differentiable Neural Network Architecture Search | ICLR-W | G | - | | Accelerating Neural Architecture Search using Performance Prediction | ICLR-W | PD | - |

2017 Venues

| Title | Venue | Type | Code | |:--------|:--------:|:--------:|:--------:| | Neural Architecture Search with Reinforcement Learning | ICLR | RL | - | | Designing Neural Network Architectures using Reinforcement Learning | ICLR | RL | - | github | | Neural Optimizer Search with Reinforcement Learning | ICML | RL | - | Large-Scale Evolution of Image Classifiers | ICML | EA | - | | Learning Curve Prediction with Bayesian Neural Networks | ICLR | PD | - | | Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization | ICLR | PD | - | | Hyperparameter Optimization: A Spectral Approach | NeurIPS-W | Other | github | | Learning to Compose Domain-Specific Transformations for Data Augmentation | NeurIPS | - | - |

Previous Venues

2012-2016

| Title | Venue | Type | Code | |:--------|:--------:|:--------:|:--------:| | Speeding up Automatic Hyperparameter Optimization of Deep Neural Networksby Extrapolation of Learning Curves | IJCAI | PD | github |

arXiv

| Title | Date | Type | Code | |:--------|:--------:|:--------:|:--------:| | NSGA-NET: A Multi-Objective Genetic Algorithm for Neural Architecture Search | 2018.10 | EA | - | | Training Frankenstein’s Creature to Stack: HyperTree Architecture Search | 2018.10 | G | - | | Population Based Training of Neural Networks | 2017.11 | EA | GitHub | | EmotionNAS: Two-stream Architecture Search for Speech Emotion Recognition | 2022.3 | G | - | | U-Boost NAS: Utilization-Boosted Differentiable Neural Architecture Search | 2022.3 | G | Github |

Awesome Surveys

| Title | Venue | Year | Code | |:--------|:--------:|:--------:|:--------:| | A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions | ACM Computing Surveys | 2021 | - | | Automated Machine Learning on Graphs: A Survey | ICLR-W | 2021 | GitHub | | On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice | Neurocomputing | 2020 |github | | AutonoML: Towards an Integrated Framework for Autonomous Machine Learning | arXiv | 2020 | - | | Automated Machine Learning | Springer Book | 2019 | - | | Neural architecture search: A survey | JMLR | 2019 | - | | AutoML: A Survey of the State-of-the-Art | arXiv | 2019 | GitHub | | A Survey on Neural Architecture Search | arXiv | 2019 | - | | Taking human out of learning applications: A survey on automated machine learning | arXiv | 2018 | - | | IoT Data Analytics in Dynamic Environments: From An Automated Machine Learning Perspective | Engineering Applications of Artificial Intelligence | 2022 |github |

Issues

Request to Add THIN-Bayes

opened on 2022-04-15 17:40:31 by swapnilsayansaha

Paper: https://www.researchgate.net/profile/Sandeep-Sandha/publication/356911955_Enabling_Hyperparameter_Tuning_of_Machine_Learning_Classifiers_in_Production/links/62171fb21857671d0d8b04f4/Enabling-Hyperparameter-Tuning-of-Machine-Learning-Classifiers-in-Production.pdf

Poster: https://cms.tinyml.org/wp-content/uploads/talks2022/Saha-S.-Sayan-SW-tools.pdf

Code: https://github.com/ARM-software/mango/tree/master/examples/THIN-Bayes

We developed a new state-of-art Bayesian NAS framework. We would be happy if you add it to your list.