Medical visual question answering based on question-type reasoning and semantic space constraint

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Computer Vision

Background:

• Medical visual question answering (Med-VQA) holds significant potential but faces challenges with diverse clinical questions and answer relationships.
• Current Med-VQA models often struggle with noise in question features and treat answers as independent classifications.

Purpose of the Study:

• To propose a novel Med-VQA framework addressing challenges in question feature extraction and answer relationship modeling.
• To improve the accuracy and robustness of Med-VQA systems for clinical applications.

Main Methods:

• A question-type reasoning module was developed to process closed-ended and open-ended questions separately using attention mechanisms.
• A semantic constraint space was designed to model relationships between candidate answers, prioritizing correlated responses.

Main Results:

• The proposed framework achieved superior performance on the VQA-RAD dataset compared to state-of-the-art methods.
• Overall accuracy reached 74.1%, with closed-ended accuracy at 82.7% (a 5.5% absolute improvement) and open-ended accuracy at 60.9%.

Conclusions:

• The novel Med-VQA framework effectively handles diverse clinical questions and leverages answer relationships for improved performance.
• This approach offers a promising advancement for accurate and reliable Med-VQA systems in medical diagnostics.