BPI-MVQA: a bi-branch model for medical visual question answering

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Natural Language Processing

Background:

• Medical visual question answering (VQA-Med) aids disease diagnosis and patient comprehension.
• A key challenge is integrating medical image semantics and question data in unlabeled datasets.

Purpose of the Study:

• To introduce a novel Bi-branched model for Medical Visual Question Answering (BPI-MVQA).
• To enhance the accuracy and understanding of medical images for VQA-Med applications.

Main Methods:

• Proposed the Bi-branched model based on Parallel networks and Image retrieval for Medical Visual Question Answering (BPI-MVQA).
• Utilized a transformer-based parallel network for complementary image feature extraction and implicit multi-modal fusion via multi-head self-attention.
• Incorporated VGG16 network for image feature extraction and retrieval of similar text descriptions.

Main Results:

• Achieved state-of-the-art results on three VQA-Med datasets.
• Main metric scores surpassed previous best results by 0.2, 1.4, and 1.1.

Conclusions:

• The BPI-MVQA model demonstrates significant effectiveness in VQA-Med.
• The bi-branch design and parallel network enable multi-angle feature extraction, improving semantic understanding and classification accuracy.
• Image retrieval assists in answering open-ended questions by leveraging image information, leading to substantial improvements in VQA-Med systems.