Exploring the performance and explainability of fine-tuned BERT models for neuroradiology protocol assignment

Area of Science:

• Artificial Intelligence
• Medical Imaging
• Natural Language Processing

Background:

• Deep learning models show promise but require careful validation in high-stakes medical applications.
• Understanding model decision-making is crucial for safe and reliable clinical implementation.
• This study focuses on Bidirectional Encoder Representations from Transformers (BERT) for medical image protocol assignment.

Purpose of the Study:

• To evaluate the performance of various pre-trained BERT models in medical image protocol classification.
• To interpret the decision-making process of BERT models in this medical context.
• To compare model reasoning with expert human judgment.

Main Methods:

• Fine-tuning four pre-trained BERT models (BERT, BioBERT, ClinicalBERT, RoBERTa) for protocol classification.
• Utilizing a gradient-based method to determine word importance for classification outputs.
• Having a radiologist review word importance scores to assess model reasoning against human logic.

Main Results:

• The standard BERT model demonstrated performance comparable to human experts on the test dataset.
• The BERT model accurately identified critical words associated with the correct medical imaging protocol.
• Analysis of misclassified instances highlighted potential systematic errors within the model's decision-making.

Conclusions:

• The BERT model shows significant potential for medical image protocol assignment, achieving near-human accuracy.
• The model's ability to identify key terms and the detection of systematic errors offer pathways for clinical refinement.
• Further development can enhance the safety and clinical utility of BERT in medical imaging workflows.