Multiway Transformer (image as a foreign language)

How this idea evolved

Each step is the idea’s introducing paper. The line between steps is the citation link between those papers.

1. 2021
   VLMoMixture of modality experts

   Shared attention with per-modality feed-forward experts, usable as a dual or fusion encoder.

   Cites · not yet reviewedcited 5× · §BEiT-3: A General-Purpose Multimodal Foundation Model

   “We use Multiway Transformers [53] as the backbone model to encode different modalities.”

2. 2022
   BEiT-3Multiway Transformer (image as a foreign language)

   Treat images as another language and pre-train one multiway model with masked 'language' modelling on all.