Domain Adaptive Transfer Learning with Specialist Models
Transfer learning is a widely used method to build high performing computer vision models. In this paper, we study the efficacy of transfer learning by examining how the choice of data impacts performance.
Also cited · not yet reviewed (3)
- JFT-300M (unreasonable effectiveness)2017 · cited 2×, 2 in Method“We use the JFT Sun2017 and ImageNet Russakovsky2015 datasets as our source pre-training data and consider a range of target datasets for fine-tuning (Section 3.2).”From this paper · §Transfer learning setup
- CNN features off-the-shelf2014 · cited 2דThe success of applying convolution neural networks to the ImageNet classification problem Krizhevsky2012 led to the finding that the features learned by a convolutional neural network perform well on a variety of image…”From this paper · §Related Work
- Do better ImageNet models transfer bette2018 · cited 2דKornblith2018 who also found that Inception v3 did slightly better than NasNet-A Zoph2017.”From this paper · §Experiments
Led to
- BiT2019 · cited 5דRather than pre-train generic representations, recent works have shown strong performance by training task-specific representations [63, 38, 61].”From BiT · §Related Work
Abstract
Transfer learning is a widely used method to build high performing computer vision models. In this paper, we study the efficacy of transfer learning by examining how the choice of data impacts performance. We find that more pre-training data does not always help, and transfer performance depends on a judicious choice of pre-training data. These findings are important given the continued increase in dataset sizes. We further propose domain adaptive transfer learning, a simple and effective pre-training method using importance weights computed based on the target dataset. Our method to compute importance weights follow from ideas in domain adaptation, and we show a novel application to transfer learning. Our methods achieve state-of-the-art results on multiple fine-grained classification datasets and are well-suited for use in practice.