Connecting Vision and Language with Localized Narratives
We propose Localized Narratives, a new form of multimodal image annotations connecting vision and language. We ask annotators to describe an image with their voice while simultaneously hovering their mouse over the region they are describing.
Also cited · not yet reviewed (1)
- The Open Images Dataset V42018 · cited 2דWe collected Localized Narratives at scale: we annotated the whole COCO [35] (123123k images), ADE20K [69] (2020k) and Flickr30k [66] (3232k) datasets, as well as 671671k images of Open Images [33].”From this paper · §Introduction
Led to
- Conceptual 12M2021 · cited 3×, 3 in Method“In addition, besides CC3M and CC12M, we also explore using the Open Images Localized Narratives dataset (LocNar) [64], as an alternative “in-domain” (from a visual standpoint) pre-training data source.”From Conceptual 12M · §Evaluating Vision-and-Language Pre-Training Data
Abstract
We propose Localized Narratives, a new form of multimodal image annotations connecting vision and language. We ask annotators to describe an image with their voice while simultaneously hovering their mouse over the region they are describing. Since the voice and the mouse pointer are synchronized, we can localize every single word in the description. This dense visual grounding takes the form of a mouse trace segment per word and is unique to our data. We annotated 849k images with Localized Narratives: the whole COCO, Flickr30k, and ADE20K datasets, and 671k images of Open Images, all of which we make publicly available. We provide an extensive analysis of these annotations showing they are diverse, accurate, and efficient to produce. We also demonstrate their utility on the application of controlled image captioning.