XRaySwinGen: Automatic medical reporting for X-ray exams with multimodal model.

Area of Science:

• Medical Imaging and Artificial Intelligence
• Computer Vision and Natural Language Processing

Background:

• Manual radiology report generation is time-consuming and prone to errors.
• Current AI in medical imaging often focuses on classification, missing crucial patient condition details.
• Generating descriptive reports from X-rays remains a challenge for AI.

Purpose of the Study:

• To develop a multimodal AI model for generating detailed, descriptive radiology reports from X-ray images.
• To overcome limitations in current AI medical image captioning by focusing on comprehensive report generation.
• To improve the accuracy and efficiency of medical reporting through advanced AI techniques.

Main Methods:

• Utilized a multimodal model combining Computer Vision (Swin Transformer) for image encoding and Natural Language Processing (GPT-2) for text decoding.
• Implemented Swin Transformer for hierarchical image representation, enhancing perception without increasing computational cost.
• Employed cross-attention layers and bilingual training (Portuguese PT-BR and English) for robust report generation.

Main Results:

• Achieved promising performance on a proposed database with ROUGE-L of 0.748 and METEOR of 0.741.
• Demonstrated effectiveness on the NIH CHEST X-ray dataset, yielding ROUGE-L of 0.404 and METEOR of 0.393.
• The model successfully generated descriptive textual reports from X-ray images.

Conclusions:

• The proposed multimodal AI model effectively generates detailed radiology reports, addressing limitations in current medical image captioning.
• The innovative use of Swin Transformer and GPT-2 offers an efficient and effective approach to AI-driven medical report generation.
• Bilingual training and promising results indicate the model's potential for improving radiology workflows and diagnostic capabilities.