Development of a large-scale medical visual question-answering dataset

Area of Science:

• Artificial Intelligence in Medicine
• Medical Imaging Analysis
• Natural Language Processing in Healthcare

Background:

• Medical Visual Question Answering (MedVQA) utilizes AI to interpret medical images, enhancing diagnostic accuracy and healthcare delivery.
• Current MedVQA approaches require refinement to better emulate human-machine interaction and integrate complex data.
• The need for advanced AI models capable of processing both visual and textual medical information is critical.

Purpose of the Study:

• To redefine Medical Visual Question Answering (MedVQA) as a generative task.
• To develop an AI model that integrates complex visual and textual information for improved medical image interpretation.
• To create a comprehensive dataset for training and evaluating MedVQA models.

Main Methods:

• Construction of the large-scale PMC-VQA dataset, comprising 227,000 VQA pairs from 149,000 medical images across diverse modalities and diseases.
• Introduction of a generative AI model that aligns visual features from a pre-trained vision encoder with a large language model.
• Initial training of the model on the PMC-VQA dataset, followed by fine-tuning on multiple public benchmarks.

Main Results:

• The developed generative model significantly outperforms existing MedVQA models in generating accurate, free-form answers.
• A manually verified, challenging test set was proposed to robustly monitor progress in generative MedVQA.
• Demonstrated superior performance in integrating visual and textual data for medical question answering.

Conclusions:

• The PMC-VQA dataset is an essential resource for advancing MedVQA research.
• The proposed generative model represents a significant breakthrough in the field of MedVQA.
• A leaderboard is maintained for comprehensive evaluation and benchmarking of state-of-the-art MedVQA approaches.