Towards a holistic framework for multimodal LLM in 3D brain CT radiology report generation

Area of Science:

• Artificial Intelligence in Medicine
• Medical Imaging Analysis
• Natural Language Processing

Background:

• Multi-modal large language models (MLLMs) are transforming healthcare, particularly in automated radiology report generation (RRG).
• Existing MLLM applications primarily focus on 2D medical images, leaving the potential for 3D image analysis largely unexplored.
• The accurate generation of diagnostic reports from 3D medical data is crucial for clinical decision-making.

Purpose of the Study:

• To develop and evaluate a model for 3D CT radiology report generation.
• To curate a novel dataset for 3D brain CT scans and corresponding reports.
• To establish a robust evaluation metric for assessing the clinical quality of generated radiology reports.

Main Methods:

• Curated the 3D-BrainCT dataset, comprising 18,885 text-scan pairs.
• Developed BrainGPT, a clinically visual instruction-tuned (CVIT) model for 3D CT RRG.
• Proposed the Feature-Oriented Radiology Task Evaluation (FORTE) metric to assess diagnostic quality.

Main Results:

• BrainGPT achieved an average FORTE F1-score of 0.71.
• Specific FORTE scores included 0.661 for degree, 0.706 for landmark, 0.693 for feature, and 0.779 for impression.
• In a Turing-like test, 74% of BrainGPT-generated reports were indistinguishable from human-written reports.

Conclusions:

• The developed framework, including dataset curation, model fine-tuning, and evaluation metrics, provides a comprehensive approach to 3D RRG.
• BrainGPT demonstrates significant potential for advancing automated radiology report generation in 3D medical imaging.
• This work aims to accelerate human-machine collaboration in next-generation healthcare through improved 3D MLLM-based RRG.