AERMNet: Attention-enhanced relational memory network for medical image report generation

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Natural Language Processing

Background:

• Automatic generation of medical image diagnostic reports aids clinicians by reducing workload and enhancing diagnostic efficiency and accuracy.
• Existing report generation models suffer from weak word correlation and lack of contextual information.

Purpose of the Study:

• To address limitations in current medical image report generation models.
• To propose a novel model that improves word correlation and contextual information utilization.

Main Methods:

• Developed an Attention-Enhanced Relational Memory Network (AERMNet) model.
• Incorporated a relational memory module updated by previously generated words to strengthen inter-word correlation.
• Utilized a double LSTM with an interaction module to minimize context loss and maximize feature information usage.

Main Results:

• AERMNet demonstrated superior performance on four medical datasets (Fetal Heart, Ultrasound, IU X-Ray, MIMIC-CXR).
• Significant improvements observed in language generation metrics, including a 2.4% Cider increase on FH, 2.4% Bleu1 on Ultrasound, 16.4% Cider on IU X-Ray, and 9.7% Bleu2 on MIMIC-CXR.
• The model generates more accurate disease information for medical image reports.

Conclusions:

• The proposed AERMNet model advances the field of medical image report generation.
• This work expands the application prospects of computer-aided diagnosis.
• The code for AERMNet is publicly available for further research and development.