Who labels best? Radiologists, rules, or large language models for CT reports on pulmonary embolism.

Area of Science:

• Artificial Intelligence in Radiology
• Natural Language Processing for Medical Reports
• Pulmonary Embolism Diagnostics

Background:

• Accurate labeling of pulmonary embolism CT reports is crucial for clinical research and quality improvement.
• Traditional methods like rule-based extractors (RBEs) have limitations in capturing complex report nuances.
• Large language models (LLMs) show promise for automating medical text analysis.

Purpose of the Study:

• To compare the performance of open-weight and proprietary LLMs, RBEs, and radiologists in labeling pulmonary embolism CT reports.
• To evaluate the effectiveness of a hybrid RBE-LLM workflow for improving labeling accuracy and efficiency.
• To assess the potential of AI in automating the extraction of structured findings from radiology reports.

Main Methods:

• A retrospective study of 2,923 structured CT reports was conducted.
• Three labeling pipelines were evaluated: RBE, LLM (open-weight and proprietary variants), and a hybrid RBE-LLM approach.
• Ground truth was established by schema matching and blinded radiologist adjudication.

Main Results:

• Top LLMs (Falcon3-10b, GPT-4.1-mini) achieved high performance (F1 0.98), surpassing the RBE (F1 0.81).
• The hybrid RBE-LLM workflow demonstrated superior performance with 99.8% accuracy and an F1 score of 0.99.
• The hybrid approach significantly reduced cohort-curation time compared to manual radiologist labeling.

Conclusions:

• Schema-constrained LLMs, both open-weight and proprietary, outperform traditional RBEs for pulmonary embolism CT report labeling.
• A hybrid RBE-LLM workflow enables highly accurate and efficient automated extraction of structured findings.
• This automated approach facilitates scalable, auditable, and consistent cohort curation for clinical research and quality improvement.