Memory-Based Cross-Modal Semantic Alignment Network for Radiology Report Generation.

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Natural Language Processing

Background:

• Automated radiology report generation aids radiologists and disease diagnosis.
• Current methods struggle to capture crucial disease information from images and reports.
• This limitation hinders the generation of fluent and accurate reports.

Purpose of the Study:

• To develop an advanced model for accurate and fluent radiology report generation.
• To address the challenge of learning latent relationships between medical images and reports.
• To improve the clinical utility of automated diagnostic tools.

Main Methods:

• Proposed a memory-based cross-modal semantic alignment model (MCSAM) using an encoder-decoder architecture.
• Incorporated a long-term clinical memory bank for disease-related representations and prior knowledge retrieval.
• Introduced a semantic alignment module (SAM) for cross-modal consistency and visual feature embedding generation.
• Utilized learnable memory tokens as prompts to enhance report generation within the decoder.

Main Results:

• MCSAM demonstrated superior performance in generating radiology reports.
• The model successfully learned latent relationships between radiology images and their corresponding reports.
• Achieved state-of-the-art results on the MIMIC-CXR dataset.

Conclusions:

• The proposed MCSAM effectively enhances automated radiology report generation.
• Memory-based learning and cross-modal semantic alignment are key to improving report accuracy and fluency.
• This approach offers a promising solution for reducing radiologist workload and aiding disease diagnosis.