A dataset for medical instructional video classification and question answering

Area of Science:

• Medical Informatics
• Computer Vision
• Natural Language Processing

Background:

• Medical videos hold significant potential for answering first aid, emergency, and educational questions.
• Existing research lacks robust systems for extracting visual answers from medical video content.

Purpose of the Study:

• Introduce novel datasets (MedVidCL and MedVidQA) and tasks (Medical Video Classification and Medical Visual Answer Localization) to advance medical video understanding.
• Foster research in cross-modal learning between medical language and video data.

Main Methods:

• Creation of MedVidCL dataset with 6,117 fine-grained annotated videos for classification.
• Development of MedVidQA dataset with 3,010 question-answer timestamp pairs from 899 videos for localization.
• Validation and correction of datasets by medical informatics experts.
• Benchmarking of proposed tasks using the created datasets.

Main Results:

• Established competitive baseline models for Medical Video Classification and Medical Visual Answer Localization.
• Demonstrated the utility of the MedVidCL and MedVidQA datasets for evaluating multimodal learning approaches.

Conclusions:

• The developed datasets and tasks provide a foundation for building sophisticated applications that can visually answer natural language questions using medical videos.
• This research facilitates the development of AI systems to enhance medical education and provide accessible health information.