Artificial Intelligence for CT and MRI Protocoling: A Meta-Analysis of Traditional Machine Learning, BERT, and Large...

BACKGROUND. Examination protocoling is a resource-intensive task. Various artificial intelligence (AI) approaches have been investigated to automate this process. OBJECTIVE. The purpose of this study was to evaluate performance of traditional machine learning (ML) models, bidirectional encoder representations from transformers (BERT) models, and large language models (LLMs) for automated CT and MRI protocoling. EVIDENCE ACQUISITION. MEDLINE, Embase, Scopus, Web of Science, IEEE Xplore, and Google Scholar databases were searched through July 2025 for studies reporting the performance of an AI-based technique in assigning protocols for CT or MRI requisitions. Accuracy results were separately extracted for all models tested in each study and pooled using a random-effects meta-analysis. AI approaches were compared using Welch t tests. Common sources of error were qualitatively summarized. EVIDENCE SYNTHESIS. The final analysis included 23 studies, comprising 1,196,259 imaging requisitions. Requisition subspecialties included body imaging (n = 4), musculoskeletal imaging (n = 3), neuroradiology (n = 6), thoracic imaging (n = 1), and multiple subspecialties (n = 9). Sixteen studies evaluated traditional ML models, eight evaluated BERT models, and five evaluated LLMs. Task-specific model fine-tuning was performed in three studies for traditional ML models, all studies for BERT models, and one study for LLMs. The overall pooled protocoling accuracy was 85% (95% CI, 83-87%). The pooled accuracy was 83% (95% CI, 80-85%) for traditional ML models, 87% (95% CI, 85-89%) for BERT models, and 86% (95% CI, 83-89%) for LLMs; these pooled accuracies were not significantly different between any pairwise combination of the three AI approaches (all p > .05). Among 30 distinct models (14 traditional ML models, nine BERT models, seven LLMs), the top-10 performing models comprised two traditional ML models, six BERT models (including the top performing model [BioBERT, a biomedical-domain BERT; accuracy, 93%]), and two LLMs. Common sources of error included ambiguous requisition text, data imbalance yielding incorrect protocol assignments for low-volume protocols, the presence of multiple clinically reasonable protocols for given requisitions, and difficulty handling requisitions containing terms strongly associated with disparate protocols. CONCLUSION. The top-performing AI models for automated CT and MRI protocoling included predominantly fine-tuned BERT models. CLINICAL IMPACT. AI tools show strong potential to help streamline radiologist workflows, possibly through hybrid AI-radiologist approaches. Fine-tuned LLMs warrant further exploration. TRIAL REGISTRATION. PROSPERO identifier CRD420251088671.