An Empirical Study of Training End-to-End Vision-and-Language Transformers
Zi-Yi Dou, Yichong Xu, Zhe Gan and 9 others
Vision-and-language (VL) pre-training has proven to be highly effective on various VL downstream tasks. While recent work has shown that fully transformer-based VL models can be more efficient than previous region-feature-based methods, their performance on downstream tasks often degrades significantly.
From the abstract
Also cited · not yet reviewed (28)
- UNITER2019 · cited 11×, 9 in Method“Vision-and-language pre-training (VLP) has now become the de facto practice to tackle these tasks tan-bansal-2019-lxmert; li2019visualbert; lu2019vilbert; su2019vl; chen2020uniter; li2020oscar.”From this paper · §Introduction
- LXMERT2019 · cited 7×, 5 in Method“Vision-and-language pre-training (VLP) has now become the de facto practice to tackle these tasks tan-bansal-2019-lxmert; li2019visualbert; lu2019vilbert; su2019vl; chen2020uniter; li2020oscar.”From this paper · §Introduction
- VinVL2021 · cited 7×, 4 in Method“Recent works kim2021vilt; xue2021probing; li2021align that tried to adopt vision transformers have not shown satisfactory performance and typically underperform state-of-the-art region-feature-based VLP models (e.g., Vin…”From this paper · §Introduction
- ViLT2021 · cited 7×, 4 in Method“This can lead to several problems: first, the object detectors are not perfect, but are usually kept frozen during VLP, which limits the capacity of the VLP models; second, it is time-consuming to extract region features…”From this paper · §Introduction
Show 24 more
- ViLBERT2019 · cited 6×, 4 in Method“Vision-and-language pre-training (VLP) has now become the de facto practice to tackle these tasks tan-bansal-2019-lxmert; li2019visualbert; lu2019vilbert; su2019vl; chen2020uniter; li2020oscar.”From this paper · §Introduction
- VisualBERT2019 · cited 6×, 4 in Method“Vision-and-language pre-training (VLP) has now become the de facto practice to tackle these tasks tan-bansal-2019-lxmert; li2019visualbert; lu2019vilbert; su2019vl; chen2020uniter; li2020oscar.”From this paper · §Introduction
- ALBEF2021 · cited 6×, 4 in Method“Recent works kim2021vilt; xue2021probing; li2021align that tried to adopt vision transformers have not shown satisfactory performance and typically underperform state-of-the-art region-feature-based VLP models (e.g., Vin…”From this paper · §Introduction
- DeiT2020 · cited 4×, 4 in Method“ViT has become a popular research topic recently dosovitskiy2020image; touvron2020deit; touvron2020deit; touvron2021going; yuan2021volo; liu2021swin; bao2021beit, and has been introduced into VLP kim2021vilt; xue2021prob…”From this paper · §The Meter Framework
- RoBERTa2019 · cited 5×, 3 in Method“Specifically, as shown in Figure 1, we dissect the model designs along multiple dimensions, including vision encoders (e.g., CLIP-ViT radford2021learning, Swin transformer liu2021swin), text encoders (e.g., RoBERTa liu20…”From this paper · §Introduction
- Swin2021 · cited 5×, 3 in Method“Transformers vaswani2017attention are prevalent in natural language processing and have recently shown promising performance in computer vision dosovitskiy2020image; liu2021swin.”From this paper · §Introduction
- ViT2020 · cited 4×, 3 in Method“Transformers vaswani2017attention are prevalent in natural language processing and have recently shown promising performance in computer vision dosovitskiy2020image; liu2021swin.”From this paper · §Introduction
- BEiT2021 · cited 4×, 3 in Method“Specifically, as shown in Figure 1, we dissect the model designs along multiple dimensions, including vision encoders (e.g., CLIP-ViT radford2021learning, Swin transformer liu2021swin), text encoders (e.g., RoBERTa liu20…”From this paper · §Introduction
- BERT2018 · cited 3×, 3 in Method“Following BERT devlin2018bert and RoBERTa liu2019roberta, VLP models tan-bansal-2019-lxmert; li2019visualbert; lu2019vilbert; su2019vl; chen2020uniter; li2020oscar first segment the input sentence into a sequence of subw…”From this paper · §The Meter Framework
- MS COCO2014 · cited 5×, 2 in Method“Vision-and-language (VL) tasks, such as visual question answering (VQA) antol2015vqa and image-text retrieval lin2014microsoft; plummer2015flickr30k, require an AI system to comprehend both the input image and text conte…”From this paper · §Introduction
- VL-BERT2019 · cited 4×, 2 in Method“Vision-and-language pre-training (VLP) has now become the de facto practice to tackle these tasks tan-bansal-2019-lxmert; li2019visualbert; lu2019vilbert; su2019vl; chen2020uniter; li2020oscar.”From this paper · §Introduction
- Oscar2020 · cited 4×, 2 in Method“Vision-and-language pre-training (VLP) has now become the de facto practice to tackle these tasks tan-bansal-2019-lxmert; li2019visualbert; lu2019vilbert; su2019vl; chen2020uniter; li2020oscar.”From this paper · §Introduction
- AdamW2017 · cited 2×, 2 in Method“For example, PixelBERT huang2020pixel and CLIP-ViL shen2021much use AdamW loshchilov2018decoupled for transformer and SGD for CNN.”From this paper · §Glossary of VLP Models
- VQA2015 · cited 3×, 1 in Method“Vision-and-language (VL) tasks, such as visual question answering (VQA) antol2015vqa and image-text retrieval lin2014microsoft; plummer2015flickr30k, require an AI system to comprehend both the input image and text conte…”From this paper · §Introduction
- Flickr30k Entities2015 · cited 3×, 1 in Method“Vision-and-language (VL) tasks, such as visual question answering (VQA) antol2015vqa and image-text retrieval lin2014microsoft; plummer2015flickr30k, require an AI system to comprehend both the input image and text conte…”From this paper · §Introduction
- VL-T52021 · cited 3×, 1 in Method“Recently, VL-T5 cho2021unifying and SimVLM wang2021simvlm, on the other hand, advocate the use of a transformer encoder-decoder architecture, where the cross-modal representations are first fed into a decoder and then to…”From this paper · §The Meter Framework
- CLIP2021 · cited 3×, 1 in Method“Specifically, as shown in Figure 1, we dissect the model designs along multiple dimensions, including vision encoders (e.g., CLIP-ViT radford2021learning, Swin transformer liu2021swin), text encoders (e.g., RoBERTa liu20…”From this paper · §Introduction
- Visual Genome2016 · cited 2×, 1 in Method“We perform the investigation by pre-training models under Meter on four commonly used image-caption datasets: COCO lin2014microsoft, Conceptual Captions sharma2018conceptual, SBU Captions ordonez2011im2text, and Visual G…”From this paper · §Introduction
- NLVR22018 · cited 2×, 1 in Method“We test them on visual question answering antol2015vqa, visual reasoning suhr2018corpus, image-text retrieval lin2014microsoft; plummer2015flickr30k, and visual entailment xie2019visual tasks.”From this paper · §Introduction
- Limits of Language Modeling2016 · cited 1×, 1 in Method“The model is trained to maximize its probability similar to noise contrastive estimation nce1; nce2.”From this paper · §The Meter Framework
- VQ-VAE2017 · cited 1×, 1 in Method“Specifically, we first use the VQ-VAE van2017neural model in DALL-E ramesh2021zero to tokenize each image into a sequence of discrete tokens.”From this paper · §The Meter Framework
- ALBERT2019 · cited 1×, 1 in Method“In this work, we study the use of BERT devlin2018bert, RoBERTa liu2019roberta, ELECTRA clark2020electra, ALBERT lan2019albert, and DeBERTa he2020deberta for text encoding.”From this paper · §The Meter Framework
- DALL·E2021 · cited 1×, 1 in Method“Specifically, we first use the VQ-VAE van2017neural model in DALL-E ramesh2021zero to tokenize each image into a sequence of discrete tokens.”From this paper · §The Meter Framework
- SimVLM2021 · cited 1×, 1 in Method“Recently, VL-T5 cho2021unifying and SimVLM wang2021simvlm, on the other hand, advocate the use of a transformer encoder-decoder architecture, where the cross-modal representations are first fed into a decoder and then to…”From this paper · §The Meter Framework
Led to
- Florence2021 · cited 5×, 3 in Method“We use METER (Dou et al. 2021) adapter to expand to fine-grained vision-language representation.”From Florence · §Approach
- Simple end-to-end captioning2022 · cited 3דTo mitigate the need for object detectors, METER (Dou et al. 2021) directly connects RoBERTa (Liu et al. 2019) and CLIP-ViT in a single model.”From Simple end-to-end captioning · §. Related Work
Abstract
Vision-and-language (VL) pre-training has proven to be highly effective on various VL downstream tasks. While recent work has shown that fully transformer-based VL models can be more efficient than previous region-feature-based methods, their performance on downstream tasks often degrades significantly. In this paper, we present METER, a Multimodal End-to-end TransformER framework, through which we investigate how to design and pre-train a fully transformer-based VL model in an end-to-end manner. Specifically, we dissect the model designs along multiple dimensions: vision encoders (e.g., CLIP-ViT, Swin transformer), text encoders (e.g., RoBERTa, DeBERTa), multimodal fusion module (e.g., merged attention vs. co-attention), architectural design (e.g., encoder-only vs. encoder-decoder), and pre-training objectives (e.g., masked image modeling). We conduct comprehensive experiments and provide insights on how to train a performant VL transformer. METER achieves an accuracy of 77.64% on the VQAv2 test-std set using only 4M images for pre-training, surpassing the state-of-the-art region-feature-based model by 1.04%, and outperforming the previous best fully transformer-based model by 1.6%. Notably, when further scaled up, our best VQA model achieves an accuracy of 80.54%. Code and pre-trained models are released at https://github.com/zdou0830/METER.