MedFuseNet: An attention-based multimodal deep learning model for visual question answering in the medical domain

Area of Science:

• Artificial Intelligence
• Medical Imaging Analysis
• Computer Vision

Background:

• Medical image interpretation requires expertise, and practitioner fatigue can lead to diagnostic errors.
• Existing visual question answering (VQA) systems are not optimized for the unique challenges of medical imaging, such as limited training data.

Purpose of the Study:

• To develop an attention-based multimodal deep learning model, MedFuseNet, for reliable visual question answering (VQA) on medical images.
• To address the challenges of limited training data and the need for interpretability in medical VQA systems.

Main Methods:

• Developed MedFuseNet, an attention-based multimodal deep learning model.
• Tackled VQA by breaking down the problem into categorization and generation tasks for answer prediction.
• Incorporated visualization of attention mechanisms to enhance model interpretability.

Main Results:

• MedFuseNet demonstrated superior performance compared to state-of-the-art VQA methods on medical image datasets.
• The model effectively handles the complexities of medical VQA with limited training data.
• Attention visualizations confirmed the interpretability of the model's predictions.

Conclusions:

• MedFuseNet provides a promising solution for AI-assisted medical diagnosis by offering a reliable 'second opinion'.
• The model's design balances performance with interpretability, crucial for clinical adoption.
• Future work can explore further enhancements for specialized medical imaging tasks.