Comparative Evaluation of Encoder- and Decoder-Based Models for Actionable Findings in CT Reports

To automate the notification workflow for actionable computed tomography (CT) reports, we investigated model configurations that maintained detection performance even with a very low positive rate. We assembled a dataset of 1000 head CT reports and compared the performance of an encoder model (ModernBERT-ja) with that of a decoder model (Llama-3-ELYZA-JP-8B). The encoder underwent supervised fine-tuning (SFT) using multiple loss functions, while the decoder was evaluated through few-shot prompting, zero-shot chain-of-thought (CoT), and SFT. We systematically decreased the positive rate in the training data to 14.4%, 10%, 5%, and 2% to assess model robustness. The encoder SFT with class-weighted focal loss achieved an F1 score of 0.870 at a positive rate of 14.4%, surpassing the decoder's best few-shot configuration (10 demonstrations, label0_first; F1 = 0.717). However, encoder performance declined substantially when the positive rate dropped below 5%. The decoder improved with only a few demonstrations, and the ordering of positive demonstrations played a crucial role in few-shot prompting. Applying SFT to the decoder raised the F1 score to 0.853, approaching that of the encoder but incurring higher training expenses. Neither architecture maintained detection capability when the positive rate was 2%. Zero-shot CoT performed less effectively than few-shot prompting under all imbalanced conditions. With sufficient positive cases, encoder SFT yields the best performance. When sampling the minority class is difficult, decoder few-shot prompting without additional training is a practical alternative. These results guide the choice between encoder fine-tuning and decoder prompting for highly imbalanced radiology-report triage of actionable findings.