Abstract:Visual question answering is fundamentally compositional in nature---a question like "where is the dog?" shares substructure with questions like "what color is the dog?" and "where is the cat?" This paper seeks to simultaneously exploit the representational capacity of deep networks and the compositional linguistic structure of questions. We describe a procedure for constructing and learning *neural module networks*, which compose collections of jointly-trained neural "modules" into deep networks for question answering. Our approach decomposes questions into their linguistic substructures, and uses these structures to dynamically instantiate modular networks (with reusable components for recognizing dogs, classifying colors, etc.). The resulting compound networks are jointly trained. We evaluate our approach on two challenging datasets for visual question answering, achieving state-of-the-art results on both the VQA natural image dataset and a new dataset of complex questions about abstract shapes.
| Comments: | Corrects an error in the evaluation of the NMN-only ablation experiment |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL); Machine Learning (cs.LG); Neural and Evolutionary Computing (cs.NE) |
| Cite as: | arXiv:1511.02799 [cs.CV] |
| (or arXiv:1511.02799v4 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.1511.02799 arXiv-issued DOI via DataCite |
Submission history
From: Jacob Andreas [view email]
[v1]
Mon, 9 Nov 2015 18:48:39 UTC (2,315 KB)
[v2]
Mon, 23 Nov 2015 06:36:22 UTC (2,315 KB)
[v3]
Wed, 1 Jun 2016 18:26:40 UTC (2,306 KB)
[v4]
Mon, 24 Jul 2017 17:15:06 UTC (2,315 KB)