Evaluation of a BERT Natural Language Processing Model for Automating CT and MRI Triage and Protocol Selection

Area of Science:

• Radiology and Medical Imaging
• Artificial Intelligence
• Natural Language Processing

Background:

• Automating the triage and protocol selection of radiology studies is crucial for efficient workflow.
• Existing machine learning models have limitations in accuracy and handling diverse imaging protocols.

Purpose of the Study:

• To evaluate the accuracy of a BERT NLP model for automating triage and protocol selection of cross-sectional imaging requisitions.
• To compare BERT's performance against Naive Bayes (NB) and Support Vector Machine (SVM) models.

Main Methods:

• A retrospective study analyzed 222,392 CT and MRI studies with 300 unique protocols.
• A BERT model was trained, validated, and tested, alongside NB and SVM comparators.
• Model performance was assessed using F1 score, precision, recall, and AUROC.

Main Results:

• BERT demonstrated superior performance in protocol selection (F1 score: 0.901) compared to SVM (0.881).
• BERT and SVM showed comparable performance for triage prediction (F1 score: ~0.844).
• BERT achieved high accuracy in multi-class predictions (up to 0.991 for protocol suggestion) and outperformed other models on minority classes.

Conclusions:

• The BERT NLP model effectively automates radiology study triage and protocol selection.
• This technology has the potential to significantly enhance radiologist workflows and streamline operations.
• Advanced NLP models are feasible for optimizing radiology department efficiency.