Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
In this paper, we present the mQA model, which is able to answer questions about the content of an image. The answer can be a sentence, a phrase or a single word.
Also cited · not yet reviewed (8)
- Learning like a Child2015 · cited 5×, 3 in Method“Similar to [25], we use the sigmoid function as the activation function of the three gates and adopt ReLU [30] as the non-linear function for the LSTM memory cells.”From this paper · §The Multimodal QA (mQA) Model
- Ask Your Neurons2015 · cited 7×, 2 in Method“There are some concurrent and independent works on this topic: [1, 23, 32]. [1] propose a large-scale dataset also based on MS COCO.”From this paper · §Related Work
- Image QA models & data2015 · cited 7×, 2 in Method“There are some concurrent and independent works on this topic: [1, 23, 32]. [1] propose a large-scale dataset also based on MS COCO.”From this paper · §Related Work
- VGG2014 · cited 2×, 1 in Method“For computer vision, methods based on Convolutional Neural Network (CNN [20]) achieve the state-of-the-art performance in various tasks, such as object classification [17, 34, 17], detection [10, 44] and segmentation [3]…”From this paper · §Related Work
Show 4 more
- GoogLeNet (Inception)2014 · cited 2×, 1 in Method“In this paper, we use the GoogleNet [36].”From this paper · §The Multimodal QA (mQA) Model
- MS COCO2014 · cited 3דThe large-scale image datasets with sentence annotations (e.g., [21, 43, 11]) play a crucial role in this progress.”From this paper · §Introduction
- Multi-World QA2014 · cited 3דThere has been recent effort on the visual question answering task [9, 2, 22, 37].”From this paper · §Related Work
- VQA2015 · cited 3דThere are some concurrent and independent works on this topic: [1, 23, 32]. [1] propose a large-scale dataset also based on MS COCO.”From this paper · §Related Work
Led to
- Visual Genome2016 · cited 8דMost new datasets Yu et al., 2015; Ren et al., 2015a; Antol et al., 2015; Gao et al., 2015 have collected QA pairs on MS-COCO images, either generated automatically by NLP tools Ren et al., 2015a or written by human work…”From Visual Genome · §Related Work
- VQA v22016 · cited 2דLanguage and vision problems such as image captioning captioning_msr; captioning_xinlei; captioning_berkeley; captioning_stanford; captioning_google; captioning_toronto; captioning_baidu_ucla and visual question answerin…”From VQA v2 · §Introduction
Abstract
In this paper, we present the mQA model, which is able to answer questions about the content of an image. The answer can be a sentence, a phrase or a single word. Our model contains four components: a Long Short-Term Memory (LSTM) to extract the question representation, a Convolutional Neural Network (CNN) to extract the visual representation, an LSTM for storing the linguistic context in an answer, and a fusing component to combine the information from the first three components and generate the answer. We construct a Freestyle Multilingual Image Question Answering (FM-IQA) dataset to train and evaluate our mQA model. It contains over 150,000 images and 310,000 freestyle Chinese question-answer pairs and their English translations. The quality of the generated answers of our mQA model on this dataset is evaluated by human judges through a Turing Test. Specifically, we mix the answers provided by humans and our model. The human judges need to distinguish our model from the human. They will also provide a score (i.e. 0, 1, 2, the larger the better) indicating the quality of the answer. We propose strategies to monitor the quality of this evaluation process. The experiments show that in 64.7% of cases, the human judges cannot distinguish our model from humans. The average score is 1.454 (1.918 for human). The details of this work, including the FM-IQA dataset, can be found on the project page: http://idl.baidu.com/FM-IQA.html