A label information fused medical image report generation framework

Area of Science:

• Artificial Intelligence
• Medical Imaging Analysis
• Natural Language Processing

Background:

• Medical imaging diagnosis reports are crucial but time-consuming and error-prone for physicians.
• Automating report generation is essential to improve efficiency and reduce diagnostic errors.
• Current methods struggle with subtle image differences and accurate abnormality description.

Purpose of the Study:

• To develop an advanced framework for automated medical imaging report generation.
• To address challenges of image similarity and accurate keyword identification in reports.
• To enhance the accuracy and clinical utility of generated diagnostic reports.

Main Methods:

• A novel framework integrating a Transformer encoder, MIX-MLP multi-label classification, co-attention mechanism (CAM) for feature fusion, and a hierarchical LSTM decoder.
• Utilizing Transformer encoder for learning long-range dependencies and extracting robust visual-semantic features.
• Employing CAM and hierarchical LSTM for precise abnormality identification and visual-text alignment.

Main Results:

• The proposed framework demonstrated superior performance over existing models on IU X-RAY and MIMIC-CXR datasets.
• Achieved improvements in both natural language generation metrics and clinical efficacy assessments.
• Successfully addressed challenges in differentiating subtle image variations and generating accurate diagnostic keywords.

Conclusions:

• The developed framework offers a significant advancement in automated medical imaging report generation.
• It effectively enhances the accuracy and reliability of diagnostic reports, aiding clinical decision-making.
• The model shows promise for widespread adoption in clinical radiology workflows.