A survey of deep-learning-based radiology report generation using multimodal inputs

Area of Science:

• Artificial Intelligence
• Medical Image Analysis
• Natural Language Processing

Background:

• Automatic radiology report generation is crucial for physician workload and resource equity.
• Current methods face challenges in integrating multi-modal data (images, clinical info, knowledge).
• Deep learning approaches, including transformers and contrastive learning, are rapidly advancing this field.

Purpose of the Study:

• To provide an up-to-date survey of deep learning techniques for automatic radiology report generation.
• To propose a general workflow encompassing data acquisition, preparation, feature learning, fusion, and report generation.
• To highlight state-of-the-art methods, large model developments, explainability, datasets, and evaluation metrics.

Main Methods:

• Summarizing recent deep learning techniques (transformers, contrastive learning, knowledge bases).
• Proposing a five-component workflow: multi-modality data acquisition, preparation, feature learning, feature fusion/interaction, and report generation.
• Reviewing large model-based methods, model explainability, public datasets, and evaluation metrics.

Main Results:

• Identified key techniques and a structured workflow for deep learning-based report generation.
• Highlighted state-of-the-art methods for each component of the proposed workflow.
• Conducted a quantitative comparison of different methods under identical experimental conditions.

Conclusions:

• The survey offers comprehensive information on multimodal data fusion for automatic radiology report generation.
• It aims to guide researchers in developing advanced algorithms for clinical report generation and medical image analysis.
• Future directions and current challenges in the field are discussed to foster further research.