Learning deep representations by mutual information estimation and maximization
In this work, we perform unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality of the input to the objective can greatly influence a representation's suitability for downstream tasks.
Also cited · not yet reviewed (1)
- CPC2018 · cited 6דHowever, when we adopted the strided crop architecture found in Oord et al. 2018, both CPC and DIM performance improved considerably.”From this paper · §Experiments
Led to
- AMDIM2019 · cited 2×, 1 in Method“We introduce a model for self-supervised representation learning based on local Deep InfoMax (Hjelm et al. 2019, DIM,).”From AMDIM · §Introduction
- MoCo2019 · cited 6×, 4 in Method“Several recent studies Wu2018a; Oord2018; Hjelm2019; Zhuang2019; Henaff2019; Tian2019; Bachman2019 present promising results on unsupervised visual representation learning using approaches related to the contrastive loss…”From MoCo · §Introduction
- SimCLR2020 · cited 2דRecent literature has attempted to relate the success of their methods to maximization of mutual information between latent representations (Oord et al. 2018; Hénaff et al. 2019; Hjelm et al. 2018; Bachman et al. 2019).”From SimCLR · §Related Work
Abstract
In this work, we perform unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality of the input to the objective can greatly influence a representation's suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and competes with fully-supervised learning on several classification tasks. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation-learning objectives for specific end-goals.