Learning to Answer Visual Questions From Web Videos.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Visual question answering (VideoQA) traditionally requires extensive manual annotation of video-question-answer pairs.
• Manual annotation is time-consuming, costly, and limits the scalability of dataset creation.

Purpose of the Study:

• To develop an automated method for generating large-scale training datasets for VideoQA.
• To address the challenge of open-vocabulary and diverse answers in VideoQA datasets.
• To introduce new evaluation benchmarks for VideoQA models.

Main Methods:

• Leveraging a question generation transformer to create question-answer pairs from transcribed video narrations.
• Generating the HowToVQA69M dataset with 69 million video-question-answer triplets.
• Employing a contrastive loss training procedure with multi-modal and answer transformers to handle diverse answers.
• Proposing zero-shot VideoQA task and VideoQA feature probe evaluation settings.

Main Results:

• Achieved excellent performance, particularly for rare answers, using the automatically generated dataset.
• Demonstrated competitive results on existing VideoQA benchmarks like MSRVTT-QA, ActivityNet-QA, MSVD-QA, and How2QA.
• Successfully generalized the dataset generation approach to other data sources, creating the WebVidVQA3M dataset.

Conclusions:

• Automated cross-modal supervision is an effective strategy for large-scale VideoQA dataset generation.
• The proposed methods enhance the training of VideoQA models, improving performance on diverse and challenging questions.
• The new datasets and evaluation settings facilitate further research in VideoQA.