Rules-Augmented GLM-5.1 Prompting for Four-Class Chest CT Protocol Selection

Area of Science:

• Radiology and Medical Imaging
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• Large language models (LLMs) are being explored for clinical applications, but their effectiveness in specific, imbalanced tasks like radiology protocol selection is not well-established.
• Classical machine learning models and human expertise are current standards for radiology protocol selection.

Purpose of the Study:

• To evaluate the performance of a rule-augmented LLM (GLM-5.1) against classical text classifiers for chest CT protocol selection.
• To determine if LLM prompting can be competitive for small, imbalanced local tasks in medical imaging.

Main Methods:

• Retrospective analysis of 755 chest CT requests from a single Canadian center.
• Comparison of GLM-5.1 with a classification-rules prompt against a majority baseline, random forest with TF-IDF and oversampling (RF-ROS), and fine-tuned BioClinicalBERT.
• Models were evaluated on stratified 70:30 held-out splits using balanced accuracy and macro-F1 metrics.

Main Results:

• GLM-5.1 achieved a balanced accuracy of 0.918 and macro-F1 of 0.838, outperforming the majority baseline and BioClinicalBERT.
• GLM-5.1's performance was not significantly different from RF-ROS (p=0.82) or BioClinicalBERT (p=0.20).
• GLM-5.1 correctly classified all 8 held-out interstitial lung disease CT cases.

Conclusions:

• Rule-augmented GLM-5.1 prompting is a feasible approach for radiology protocol selection in narrow, single-center tasks.
• LLM performance was comparable to imbalance-aware classical machine learning methods.
• Findings suggest local rule encoding is a viable adaptation strategy for LLMs, rather than a complete replacement for classical ML or human oversight.