MPCM-RRG: Multi-modal Prompt Collaboration Mechanism for Radiology Report Generation

Area of Science:

• Artificial Intelligence
• Medical Imaging Analysis
• Natural Language Processing

Background:

• Medical report generation from images aids physicians but current Transformer models overlook confounding factors and data imbalance.
• Existing models struggle to accurately describe abnormalities due to skewed normal/abnormal content in reports.

Purpose of the Study:

• To propose a novel Multi-modal Prompt Collaboration Mechanism for Radiology Report Generation Model (MPCM-RRG).
• To enhance the accuracy and efficiency of automated medical report generation by addressing limitations in existing Transformer-based models.

Main Methods:

• MPCM-RRG incorporates a Visual Causal Prompting Module (VCP) using chest X-ray masks and causal inference to minimize irrelevant region influence.
• A Textual Prompt-Guided Feature Enhancement Module (TPGF) addresses text imbalance and focuses on lesion areas via multi-head attention.
• A Visual-Textual Semantic Consistency Module (VTSC) aligns visual and textual representations using contrastive consistency loss.

Main Results:

• The proposed MPCM-RRG model demonstrated superior performance compared to existing methods on the IU X-ray and MIMIC-CXR datasets.
• The model effectively captures critical lesion information by minimizing confounding factors and addressing data imbalance.

Conclusions:

• MPCM-RRG significantly improves the quality of generated medical reports by leveraging multi-modal prompt collaboration.
• The model's causal inference and feature enhancement approach offers a promising direction for advancing automated medical report generation.