S3-Net: A Self-Supervised Dual-Stream Network for Radiology Report Generation.

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Natural Language Processing

Background:

• Automated radiology report generation aims to reduce radiologist workload.
• Existing methods often overlook long-range visual dependencies and cross-modal alignment.

Purpose of the Study:

• To propose a novel end-to-end model, the Self-Supervised dual-Stream Network (S3-Net), for enhanced radiology report generation.
• To address limitations in visual feature extraction and cross-modal mapping in previous approaches.

Main Methods:

• Developed a Dual-Stream Visual Feature Extractor (DSVFE) using ResNet and SwinTransformer to capture both local and long-range visual dependencies.
• Introduced a Fusion Alignment Module (FAM) to integrate dual-stream visual features and align them with textual information.
• Incorporated Self-Supervised Learning with Mask (SSLM) to improve visual feature representation.

Main Results:

• The S3-Net model demonstrated superior performance on the IU X-ray and MIMIC-CXR datasets.
• Achieved improved language generation metrics compared to existing state-of-the-art models.
• The proposed DSVFE and FAM modules effectively captured and aligned multi-modal information.

Conclusions:

• The S3-Net model offers a significant advancement in automated radiology report generation.
• Effective integration of local and global visual features, along with cross-modal alignment, is crucial for accurate report generation.
• The self-supervised learning approach further enhances the model's representational capabilities.