Medical image captioning via generative pretrained transformers

Area of Science:

• Artificial Intelligence in Medicine
• Medical Imaging Analysis
• Natural Language Processing

Background:

• Clinical image interpretation relies heavily on accurate and descriptive reports.
• Generating these reports manually is time-consuming and prone to variability.
• Integrating radiological scans with patient data can enhance diagnostic accuracy.

Purpose of the Study:

• To develop an automated model for clinical image caption generation.
• To combine radiological scan analysis with structured patient information.
• To produce comprehensive radiology records with pathology localization.

Main Methods:

• A novel model integrating Show-Attend-Tell and GPT-3 language models was proposed.
• The model analyzes radiological scans and textual patient records.
• 2D heatmaps were generated to localize identified pathologies.

Main Results:

• The model successfully generated descriptive textual summaries of clinical images.
• Pathology identification, location, and visual localization via heatmaps were achieved.
• Effective applicability to chest X-ray image captioning was demonstrated on Open-I and MIMIC-CXR datasets.

Conclusions:

• The proposed model offers an efficient approach to automatic radiology report generation.
• Combining image and text analysis enhances the comprehensiveness of clinical captions.
• This technology shows promise for improving medical documentation and diagnostic support.