RIHA: Report-Image Hierarchical Alignment for Radiology Report Generation.

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Natural Language Processing

Background:

• Radiology report generation (RRG) aims to automate diagnostic report creation from medical images, reducing radiologist workload and errors.
• Current RRG methods often fail to capture the hierarchical structure of reports, treating them as flat sequences, which limits alignment accuracy.
• Achieving fine-grained alignment between complex visual features and the semantic hierarchy of radiology reports remains a significant challenge.

Purpose of the Study:

• To propose a novel end-to-end framework, RIHA (Report-Image Hierarchical Alignment Transformer), for precise multi-level alignment between radiological images and reports.
• To enhance the accuracy and clinical efficacy of automated radiology report generation by addressing the limitations of existing methods.
• To improve cross-modal mapping for capturing nuanced semantics in clinical narratives.

Main Methods:

• RIHA employs a hierarchical alignment strategy across paragraph, sentence, and word levels.
• Introduces a Visual Feature Pyramid (VFP) for multi-scale visual feature extraction and a Text Feature Pyramid (TFP) for multi-granularity textual structure representation.
• Utilizes a Cross-modal Hierarchical Alignment (CHA) module with optimal transport for feature alignment and Relative Positional Encoding (RPE) in the decoder for enhanced token-level alignment.

Main Results:

• RIHA demonstrates superior performance compared to state-of-the-art models on benchmark chest X-ray datasets (IU-Xray and MIMIC-CXR).
• The proposed hierarchical alignment significantly improves natural language generation metrics for radiology reports.
• The framework shows enhanced clinical efficacy in generating accurate and contextually relevant diagnostic reports.

Conclusions:

• RIHA offers a significant advancement in radiology report generation by effectively addressing the challenge of hierarchical cross-modal alignment.
• The multi-level alignment strategy enables more precise mapping of visual information to textual descriptions, leading to improved report quality.
• This hierarchical approach holds promise for more reliable and accurate automated diagnostic reporting in clinical practice.