BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers
Masked image modeling (MIM) has demonstrated impressive results in self-supervised representation learning by recovering corrupted image patches. However, most existing studies operate on low-level image pixels, which hinders the exploitation of high-level semantics for representation models.
Also cited · not yet reviewed (9)
- VQ-VAE2017 · cited 4×, 3 in Method“VQ-VAE (van den Oord et al. 2017) converts an image into a sequence of discrete codes and then reconstructs the input image based on discrete codes.”From this paper · §Related Work
- BEiT2021 · cited 10×, 2 in Method“BEiT v2 inherits the masked image modeling framework defined by BEiT (Bao et al. 2022), which uses a visual tokenizer to convert each image to a set of discrete visual tokens.”From this paper · §Methodology
- ViT-VQGAN2021 · cited 4×, 2 in Method“where j∈{1,2,⋯,K}j\in\{1,2,\cdots,K\} and ℓ2\ell_{2} normalization is used for codebook lookup (Yu et al. 2021).”From this paper · §Methodology
- CLIP2021 · cited 3×, 1 in Method“The output vectors {𝒐i}i=1N\{{\bm{o}}_{i}\}_{i=1}^{N} aim at reconstructing the semantic features of a teacher model, e.g., DINO (Caron et al. 2021), and CLIP (Radford et al. 2021).”From this paper · §Methodology
Show 5 more
- ViT2020 · cited 2×, 1 in Method“The vision Transformers (ViTs; Dosovitskiy et al. 2020) are employed as the backbone networks to obtain image representations.”From this paper · §Methodology
- MAE2021 · cited 4דAs shown in Table 3, compared with MAE (He et al. 2022), BEiT v2 achieves dramatic gains across datasets, demonstrating the superiority of the proposed method in terms of model generalization.”From this paper · §Experiments
- BERT2018 · cited 2דThe MIM method has achieved great success in language task (Devlin et al. 2019; Dong et al. 2019; Bao et al. 2020).”From this paper · §Related Work
- UniLM2019 · cited 2דThe MIM method has achieved great success in language task (Devlin et al. 2019; Dong et al. 2019; Bao et al. 2020).”From this paper · §Related Work
- DALL·E2021 · cited 2דDALL-E (Ramesh et al. 2021) uses the Gumbel-softmax relaxation for quantization instead of the nearest neighbor lookup in VQ-VAE.”From this paper · §Related Work
Led to
- VL-BEiT2022 · cited 1×, 1 in Method“We use image tokenizer of BEiTv2 [26] to obtain the discrete tokens as the reconstructed targets.”From VL-BEiT · §Methods
- BEiT-32022 · cited 4×, 2 in Method“Second, the pretraining task based on masked data modeling has been successfully applied to various modalities, such as texts [14], images [3, 40], and image-text pairs [7].”From BEiT-3 · §Introduction: The Big Convergence
- EVA2022 · cited 10דHowever, there remains a debate that (i) tokenized semantic features could provide better supervision signal for masked modeling in vision bao2021beit; beitv2; beit3, and (ii) good performances could be also achieved via…”From EVA · §Introduction
Abstract
Masked image modeling (MIM) has demonstrated impressive results in self-supervised representation learning by recovering corrupted image patches. However, most existing studies operate on low-level image pixels, which hinders the exploitation of high-level semantics for representation models. In this work, we propose to use a semantic-rich visual tokenizer as the reconstruction target for masked prediction, providing a systematic way to promote MIM from pixel-level to semantic-level. Specifically, we propose vector-quantized knowledge distillation to train the tokenizer, which discretizes a continuous semantic space to compact codes. We then pretrain vision Transformers by predicting the original visual tokens for the masked image patches. Furthermore, we introduce a patch aggregation strategy which associates discrete image patches to enhance global semantic representation. Experiments on image classification and semantic segmentation show that BEiT v2 outperforms all compared MIM methods. On ImageNet-1K (224 size), the base-size BEiT v2 achieves 85.5% top-1 accuracy for fine-tuning and 80.1% top-1 accuracy for linear probing. The large-size BEiT v2 obtains 87.3% top-1 accuracy for ImageNet-1K (224 size) fine-tuning, and 56.7% mIoU on ADE20K for semantic segmentation. The code and pretrained models are available at https://aka.ms/beitv2.