Automated classification of brain MRI reports using fine-tuned large language models

Area of Science:

• Artificial Intelligence in Radiology
• Medical Natural Language Processing
• Brain MRI Analysis

Background:

• Accurate classification of brain Magnetic Resonance Imaging (MRI) reports is crucial for treatment planning.
• Manual classification by radiologists is time-consuming and subject to variability.
• Large Language Models (LLMs) show potential for automating complex medical text analysis.

Purpose of the Study:

• To evaluate the efficacy of fine-tuned LLMs in classifying brain MRI reports.
• To compare the performance of LLMs against human radiologists in this classification task.
• To assess the time efficiency of LLM-based classification.

Main Methods:

• A retrospective study utilized a large dataset of brain MRI reports (759 training, 284 validation, 164 test).
• A Bidirectional Encoder Representations from Transformers (BERT) Japanese model was fine-tuned for classification into nontumor, posttreatment tumor, and pretreatment tumor categories.
• Model performance was evaluated against two independent radiologists on the test dataset.

Main Results:

• The fine-tuned LLM achieved an overall accuracy of 0.970, with high sensitivity and specificity across all groups.
• No statistically significant differences in accuracy, sensitivity, or specificity were observed between the LLM and human readers (p ≥ 0.371).
• The LLM classified reports 20-26 times faster than radiologists.

Conclusions:

• Fine-tuned LLMs demonstrate performance comparable to radiologists in classifying brain MRI reports.
• LLMs offer a substantial time-saving advantage for this diagnostic task.
• This technology holds promise for improving efficiency in radiological workflow.