Deep Fragment Embeddings for Bidirectional Image Sentence Mapping
We introduce a model for bidirectional retrieval of images and sentences through a multi-modal embedding of visual and natural language data. Unlike previous models that directly map images or sentences into a common embedding space, our model works on a finer level and embeds fragments of images (objects) and fragments of sentences (typed dependency tree relations) into a common space.
Deep Fragment Embeddings has no earlier papers in this dataset.
Led to
- Karpathy visual-semantic alignment2014 · cited 10×, 5 in Method“We build on the approach of Karpathy et al. defrag, who learn to ground dependency tree relations to image regions with a ranking objective.”From Karpathy visual-semantic alignment · §Our Model
- Flickr30k Entities2015 · cited 4דWe build on the Flickr30k dataset (Young et al., 2014), a popular benchmark for caption generation and retrieval that has been used, among others, by Chen and Zitnick, 2015; Donahue et al., 2015; Fang et al., 2015; Gong…”From Flickr30k Entities · §Introduction
Abstract
We introduce a model for bidirectional retrieval of images and sentences through a multi-modal embedding of visual and natural language data. Unlike previous models that directly map images or sentences into a common embedding space, our model works on a finer level and embeds fragments of images (objects) and fragments of sentences (typed dependency tree relations) into a common space. In addition to a ranking objective seen in previous work, this allows us to add a new fragment alignment objective that learns to directly associate these fragments across modalities. Extensive experimental evaluation shows that reasoning on both the global level of images and sentences and the finer level of their respective fragments significantly improves performance on image-sentence retrieval tasks. Additionally, our model provides interpretable predictions since the inferred inter-modal fragment alignment is explicit.