Data-Efficient Language Model for Assessing Pulmonary Embolism Diagnostic Certainty From Radiology Reports: Model...

Area of Science:

• Medical Imaging Analysis
• Natural Language Processing in Healthcare
• Radiology Report Interpretation

Background:

• Computed tomography pulmonary angiography (CTPA) is standard for diagnosing pulmonary embolism (PE).
• Diagnostic uncertainty in CTPA interpretation is common, leading to inconsistent clinical decisions.
• Technical limitations and ambiguous language contribute to interpretation challenges.

Purpose of the Study:

• To develop a prompt-free, data-efficient method for assessing PE diagnostic certainty in CTPA reports.
• To evaluate the performance of small pretrained language models against advanced large language models (LLMs).
• To compare a novel model, PECertainty, with existing prompt-free and prompt-dependent methods.

Main Methods:

• Developed PECertainty, a lightweight, prompt-free model, using 173 CTPA reports annotated by radiologists.
• Compared PECertainty with baseline prompt-free methods (SVM, Random Forest, RoBERTa) and LLM-based methods (Gemma, Llama, GPT-3.5).
• Conducted sensitivity analyses with limited training data (1-10 examples/category) and external validation on 420 reports.

Main Results:

• PECertainty demonstrated comparable or superior performance to top prompt-dependent LLMs and all prompt-free baselines.
• In few-shot settings (10 examples/category), PECertainty (F1-score 0.80) outperformed GPT-3.5 fine-tuning (0.74) and in-context learning (0.65).
• External validation showed good generalization (F1-score 0.77); however, radiologists found PECertainty less interpretable than fine-tuned GPT-3.5.

Conclusions:

• PECertainty provides accurate, data-efficient assessment of diagnostic certainty from free-text CTPA reports, especially in low-resource settings.
• As an open-source alternative to proprietary LLMs, PECertainty can enhance communication between radiologists and referring physicians.
• Improving model interpretability is a key future direction for PECertainty.