Uncertainty-Aware Cross-Modal Retrieval for Medical Report Generation

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Computer Vision

Background:

• Current retrieval-augmented medical report generation (MRG) methods struggle with single-modal reliance, limiting semantic capture and cross-modal alignment.
• Existing systems lack reliable information control, leading to irrelevant content and potential hallucinations in generated reports.

Purpose of the Study:

• To propose Uncertainty-aware Cross-modal Alignment and Refinement (U-CAR), a unified framework to enhance semantic integration and retrieval reliability in MRG.
• To address limitations of single-modal retrieval and information control in automatic medical report generation.

Main Methods:

• U-CAR employs a cross-modal alignment module for fine-grained visual-textual correspondences, guiding dual-path retrieval-aware memory banks.
• A cross-modal retrieval-augmented generation strategy jointly attends to visual and textual context for enriched semantic coverage.
• An uncertainty-aware refinement mechanism adaptively controls retrieval necessity based on generation confidence.

Main Results:

• U-CAR achieved a 9% improvement in CIDEr on the IU X-Ray dataset.
• U-CAR demonstrated a 4% gain in BLEU-4 on the MIMIC-CXR dataset.
• The framework significantly outperforms current state-of-the-art methods in generating accurate and coherent medical reports.

Conclusions:

• U-CAR effectively enhances semantic integration and retrieval reliability for automatic medical report generation.
• The proposed framework generates clinically relevant, accurate, and coherent medical reports by leveraging multimodal information.
• U-CAR represents a significant advancement in addressing challenges in retrieval-augmented MRG.