Eye Gaze Guided Cross-Modal Alignment Network for Radiology Report Generation

Area of Science:

• Medical Imaging and Artificial Intelligence
• Natural Language Processing in Healthcare
• Radiology and Clinical Decision Support

Background:

• Automatic radiology report generation aims to improve diagnostic accuracy and efficiency.
• Current data-driven methods struggle with incorporating essential medical prior knowledge.
• Challenges exist in aligning radiology images with reports, especially concerning prior knowledge.

Purpose of the Study:

• To develop an advanced network for accurate medical report generation.
• To integrate radiologists' eye gaze data as prior knowledge for improved report fidelity.
• To enhance the alignment between visual and textual information in radiology reports.

Main Methods:

• Introduced the Eye Gaze Guided Cross-modal Alignment Network (EGGCA-Net).
• Employed a Dual Fine-Grained Branch (DFGB) and Multi-Task Branch (MTB) for multi-level semantic alignment.
• Incorporated Sentence Fine-grained Prototype Module (SFPM) and Multi-task Feature Fusion Module (MFFM).
• Utilized a label matching mechanism for disease state consistency.

Main Results:

• The EGGCA-Net demonstrated superior performance compared to existing methods.
• Enhanced accuracy and comprehensibility in generated radiology reports.
• Achieved improved results on the Open-i and MIMIC-CXR benchmark datasets.

Conclusions:

• The proposed EGGCA-Net effectively leverages eye gaze prior knowledge for better report generation.
• The network successfully addresses limitations of previous data-driven approaches.
• This method shows significant potential for advancing automated clinical diagnosis support.