Automated Clinical Information Extraction from Diagnostic and Nondiagnostic Radiology Reports Using Modern Language...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Radiology Reporting

Background:

• Automating clinical diagnosis extraction from unstructured radiology reports is challenging.
• Venous thromboembolism (VTE) diagnosis relies on imaging, but reports can be nondiagnostic.
• Integrating structured Electronic Health Record (EHR) data can aid diagnostic adjudication.

Purpose of the Study:

• To automate clinical diagnosis extraction from both diagnostic and nondiagnostic radiology reports.
• To evaluate the performance of large language models (LLMs) and BERT models for VTE diagnosis.
• To improve the management of diagnostic uncertainty in automated information extraction.

Main Methods:

• Extracted radiology reports (venous duplex, CT, V/Q scans) from EHR data (2011-2020).
• Compared multiple LLMs and BERT models for multiclass classification of VTE diagnoses.
• Utilized ICD-10 codes and anticoagulation data to adjudicate VTE in cases with nondiagnostic reports.

Main Results:

• The Llama-3.3 model achieved high accuracy, detecting 95% of VTE-positive reports and 87% of nondiagnostic reports.
• Area under the curve metrics ranged from 0.83 to 0.96 for ROC and 0.57 to 0.94 for precision-recall across models.
• Integrating structured EHR data and minimal chart review increased positive detection rate to 98% for VTE.

Conclusions:

• LLMs, particularly Llama-3.3, demonstrate high sensitivity and specificity for VTE diagnosis and identifying nondiagnostic reports.
• Combining LLMs with structured EHR data and limited chart review effectively manages diagnostic uncertainty.
• Automated information extraction from radiology reports can be significantly enhanced by this integrated approach.