Learning Transferable Architectures for Scalable Image Recognition
Developing neural network image classification models often requires significant architecture engineering. In this paper, we study a method to learn the model architectures directly on the dataset of interest.
Also cited · not yet reviewed (8)
- Inception v32015 · cited 9×, 2 in Method“Starting from the seminal work of krizhevsky2012imagenet on using convolutional architectures fukushima1982neocognitron; lecun1998gradient for ImageNet deng2009imagenet classification, successive advancements through arc…”From this paper · §Introduction
- GoogLeNet (Inception)2014 · cited 5×, 2 in Method“Starting from the seminal work of krizhevsky2012imagenet on using convolutional architectures fukushima1982neocognitron; lecun1998gradient for ImageNet deng2009imagenet classification, successive advancements through arc…”From this paper · §Introduction
- ResNet2015 · cited 5×, 2 in Method“Starting from the seminal work of krizhevsky2012imagenet on using convolutional architectures fukushima1982neocognitron; lecun1998gradient for ImageNet deng2009imagenet classification, successive advancements through arc…”From this paper · §Introduction
- VGG2014 · cited 4×, 1 in Method“Starting from the seminal work of krizhevsky2012imagenet on using convolutional architectures fukushima1982neocognitron; lecun1998gradient for ImageNet deng2009imagenet classification, successive advancements through arc…”From this paper · §Introduction
Show 4 more
- MobileNets2017 · cited 5דNotably, the smallest version of NASNet achieves 74.0% top-1 accuracy on ImageNet, which is 3.1% better than previously engineered architectures targeted towards mobile and embedded vision tasks howard2017mobilenets; shu…”From this paper · §Introduction
- BatchNorm2015 · cited 3דThanks to this property of the cells, we can generate a family of models that achieve accuracies superior to all human-invented models at equivalent or smaller computational budgets szegedy2016rethinking; BatchNorm.”From this paper · §Introduction
- Faster R-CNN2015 · cited 2דIn our experiments, the features learned by NASNets from ImageNet classification can be combined with the Faster-RCNN framework faster_rcnn to achieve state-of-the-art on COCO object detection task for both the largest a…”From this paper · §Introduction
- ResNeXt2016 · cited 2דStarting from the seminal work of krizhevsky2012imagenet on using convolutional architectures fukushima1982neocognitron; lecun1998gradient for ImageNet deng2009imagenet classification, successive advancements through arc…”From this paper · §Introduction
Led to
- MobileNetV22018 · cited 2דRecently, LearningToLearnScale; GeneticCNN; EvolutionImageClassifiers; NAS_reinforcement, opened up a new direction of bringing optimization methods including genetic algorithms and reinforcement learning to architectura…”From MobileNetV2 · §Related Work
- MnasNet2018 · cited 13×, 5 in Method“As modern CNN models become increasingly deeper and larger inceptionresnet17; senet18; nas_imagenet18; amoebanets18, they also become slower, and require more computation.”From MnasNet · §Introduction
- EfficientNet2019 · cited 4×, 2 in Method“Table 5 shows the transfer learning performance: (1) Compared to public available models, such as NASNet-A (Zoph et al. 2018) and Inception-v4 (Szegedy et al. 2017), our EfficientNet models achieve better accuracy with 4…”From EfficientNet · §Experiments
Abstract
Developing neural network image classification models often requires significant architecture engineering. In this paper, we study a method to learn the model architectures directly on the dataset of interest. As this approach is expensive when the dataset is large, we propose to search for an architectural building block on a small dataset and then transfer the block to a larger dataset. The key contribution of this work is the design of a new search space (the "NASNet search space") which enables transferability. In our experiments, we search for the best convolutional layer (or "cell") on the CIFAR-10 dataset and then apply this cell to the ImageNet dataset by stacking together more copies of this cell, each with their own parameters to design a convolutional architecture, named "NASNet architecture". We also introduce a new regularization technique called ScheduledDropPath that significantly improves generalization in the NASNet models. On CIFAR-10 itself, NASNet achieves 2.4% error rate, which is state-of-the-art. On ImageNet, NASNet achieves, among the published works, state-of-the-art accuracy of 82.7% top-1 and 96.2% top-5 on ImageNet. Our model is 1.2% better in top-1 accuracy than the best human-invented architectures while having 9 billion fewer FLOPS - a reduction of 28% in computational demand from the previous state-of-the-art model. When evaluated at different levels of computational cost, accuracies of NASNets exceed those of the state-of-the-art human-designed models. For instance, a small version of NASNet also achieves 74% top-1 accuracy, which is 3.1% better than equivalently-sized, state-of-the-art models for mobile platforms. Finally, the learned features by NASNet used with the Faster-RCNN framework surpass state-of-the-art by 4.0% achieving 43.1% mAP on the COCO dataset.