Comparative Evaluation of Encoder- and Decoder-Based Models for Actionable Findings in CT Reports.

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Natural Language Processing

Background:

• Automating the notification workflow for actionable computed tomography (CT) reports is crucial for timely clinical intervention.
• Highly imbalanced datasets, common in medical reporting, pose significant challenges for machine learning model performance.

Purpose of the Study:

• To investigate and compare the performance of encoder and decoder models in detecting actionable findings in head CT reports under varying degrees of data imbalance.
• To guide the selection of appropriate AI models for the triage of radiology reports with actionable findings.

Main Methods:

• A dataset of 1000 head CT reports was used to compare a supervised fine-tuned (SFT) encoder model (ModernBERT-ja) with a decoder model (Llama-3-ELYZA-JP-8B) evaluated via few-shot prompting, zero-shot CoT, and SFT.
• The positive rate in the training data was systematically reduced to assess model robustness (14.4% down to 2%).
• Performance was evaluated using the F1 score, with specific attention to class-weighted focal loss for the encoder and demonstration ordering for the decoder.

Main Results:

• Encoder SFT with class-weighted focal loss achieved an F1 score of 0.870 at a 14.4% positive rate, outperforming the decoder's best few-shot approach (F1=0.717).
• Encoder performance significantly degraded below a 5% positive rate, while the decoder showed improvement with few-shot prompting, sensitive to demonstration order.
• SFT improved the decoder's F1 score to 0.853 but increased training costs; neither model maintained capability at a 2% positive rate.

Conclusions:

• Encoder SFT is optimal for actionable finding detection when sufficient positive cases are available.
• Decoder few-shot prompting presents a viable, low-training alternative for highly imbalanced datasets where minority class sampling is challenging.
• Model selection depends on data availability and the specific requirements of radiology report triage for actionable findings.