From segmentation to explanation: Generating textual reports from MRI with LLMs

Area of Science:

• Medical Imaging and Artificial Intelligence
• Natural Language Processing in Healthcare

Background:

• Deep learning models in medical imaging lack transparency, hindering clinician trust in AI diagnoses.
• Explainable AI (XAI) is crucial for integrating AI into clinical practice and ensuring reliable healthcare outcomes.

Purpose of the Study:

• To develop a novel framework for enhancing AI explainability in medical imaging.
• To generate comprehensive, human-readable medical reports from AI analyses using Large Language Models (LLMs).

Main Methods:

• Integration of semantic segmentation models with atlas-based mapping and LLMs for report generation.
• Implementation of an anti-hallucination design using structured JSON and prompt constraints to ensure factual accuracy.
• Validation of the framework on brain tumor (glioma) and multiple sclerosis lesion detection tasks.

Main Results:

• High segmentation accuracy achieved with the SegResNet model.
• LLMs (Gemma, Llama, Mistral) demonstrated effectiveness in generating diverse and informative explanatory reports.
• Generated reports were evaluated for lexical diversity, readability, coherence, and information coverage.

Conclusions:

• The proposed method significantly enhances the transparency and interpretability of AI in medical imaging.
• The framework's generalizability was validated across different medical imaging scenarios, increasing trust in AI applications.
• Publicly available code and examples facilitate the adoption and further development of explainable AI in healthcare.