A Language-Guided Progressive Fusion Network with semantic density alignment for Medical Visual Question Answering

Shuxian Du ¹, Shuang Liang ¹, Yu Gu ¹

⁰School of Biomedical Engineering, Capital Medical University, Beijing, China; Laboratory for Clinical Medicine, Capital Medical University, Beijing, China; Beijing Key Laboratory of Fundamicationental Research on Biomechanics in Clinical Application, Capital Medical University, Beijing, China.

Journal of Biomedical Informatics +

|

March 20, 2025

View abstract on PubMed

Summary

A new Language-Guided Progressive Fusion Network (LGPFN) enhances Medical Visual Question Answering (Med-VQA) by addressing data inconsistencies and long-tail distributions. This AI approach improves accuracy across multiple datasets, aiding universal health coverage.

Area Of Science

Artificial Intelligence
Medical Imaging
Natural Language Processing

Background

Medical Visual Question Answering (Med-VQA) is crucial for resource-limited settings.
Existing Med-VQA models struggle with inconsistent information density and dataset biases.
Addressing these limitations is key to advancing Med-VQA capabilities.

Purpose Of The Study

To propose a novel Language-Guided Progressive Fusion Network (LGPFN) for Med-VQA.
To overcome challenges posed by information density disparities and long-tail data distributions.
To enhance the performance and generalizability of Med-VQA systems.

Main Methods

Developed the Language-Guided Progressive Fusion Network (LGPFN).
Implemented Question-Guided Progressive Multimodal Fusion (QPMF) for feature integration.
Utilized a Language-Gate Mechanism (LGM) for sample classification (Closed-Ended/Open-Ended).
Employed Triple Semantic Feature Alignment (TriSFA) for optimizing answer prediction.

Main Results

Achieved state-of-the-art performance on multiple Med-VQA datasets.
Obtained top accuracies: 80.39% (VQA-RAD), 84.07% (SLAKE), 75.74% (PathVQA), 70.60% (VQA-Med 2019).
Demonstrated the effectiveness and generalizability of the LGPFN framework.

Conclusions

The LGPFN model effectively addresses key challenges in Med-VQA.
The proposed framework shows significant potential for medical AI applications.
This advancement could contribute to improving universal health coverage through AI-powered medical insights.

Keywords:

Guided attention Language-Gated mechanism Medical visual question answering Triple semantic feature alignment