Memory Guided Transformer With Spatio-Semantic Visual Extractor for Medical Report Generation

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Radiology

Background:

• Medical imaging report generation is time-consuming and prone to errors, especially for inexperienced radiologists.
• Automatic reporting systems aim to improve diagnostic accuracy and efficiency.
• Transformer models show promise for report generation but struggle with capturing detailed image features.

Purpose of the Study:

• To develop an improved method for automatic radiology report generation.
• To enhance transformer models' ability to extract spatial and semantic information from medical images.
• To enable more detailed and accurate radiology reports.

Main Methods:

• Proposed a spatio-semantic visual extractor (SSVE) integrated into a ResNet 101 backbone.
• Incorporated a deformable network for spatially invariant features and a semantic network for multi-scale semantic information.
• Fused network representations to capture fine-grained image details.

Main Results:

• The proposed SSVE model demonstrated superior performance compared to existing methods.
• The model effectively captures multi-scale spatial and semantic information from radiology images.
• Enhanced detail in generated reports leading to improved diagnostic potential.

Conclusions:

• The SSVE significantly improves the quality and accuracy of transformer-based medical report generation.
• This approach addresses limitations in capturing fine-grained details in current models.
• The method holds potential for enhancing diagnostic efficiency and accuracy in radiology.