Focal Visual-Text Attention for Memex Question Answering.

Area of Science:

• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Neural networks excel at single-image visual question answering.
• Real-world applications require analyzing multimedia collections, like photo albums, with sequential data.

Purpose of the Study:

• To introduce the MemexQA task for multimodal question answering on sequential photo collections.
• To develop a model that answers questions and identifies grounding photos to aid memory recall from personal photo albums.

Main Methods:

• Presented the MemexQA dataset, the first multimodal QA dataset from personal photo albums.
• Proposed an end-to-end trainable hierarchical network for dynamic focus on sequential media and time.
• Utilized a hierarchical process to dynamically focus on relevant media and time within sequential data.

Main Results:

• The proposed model outperforms strong baselines on the MemexQA dataset.
• The model successfully identifies the most relevant grounding photos for answer justification.
• Demonstrated effective performance on the challenging task of memory recovery from photo sequences.

Conclusions:

• The MemexQA task and dataset enable new research in multimodal, sequential question answering.
• The proposed hierarchical network is effective for memory recovery from personal photo collections.
• This work advances the ability of AI to understand and retrieve information from sequential visual data.