NMN-VD: A Neural Module Network for Visual Dialog

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Visual dialog systems face challenges with visual grounding and coreference resolution.
• Existing models struggle to accurately link textual references to visual elements.

Purpose of the Study:

• To propose a novel neural module network for visual dialog (NMN-VD) to address visual grounding and coreference resolution.
• To enhance the performance of multimodal AI in visual dialog tasks.

Main Methods:

• Developed a question-customized modular network (NMN-VD) that dynamically selects relevant modules.
• Introduced a 'Refer' module to resolve pronoun-based visual coreference using a reference pool.
• Incorporated a 'Compare' module for handling comparative questions and a 'Find' module with triple-attention for visual grounding.

Main Results:

• The NMN-VD model effectively resolves visual coreference resolution problems, including impersonal pronouns.
• The 'Find' module successfully addresses visual grounding issues between questions and images.
• Experiments on large-scale benchmark data demonstrate the efficacy and high performance of NMN-VD.

Conclusions:

• The proposed NMN-VD model significantly improves visual dialog capabilities by tackling key challenges.
• NMN-VD offers an efficient and effective approach to multimodal AI for complex visual dialog tasks.