Automated risk scoring for venous thromboembolism using large language models with expert knowledge-augmented...

Area of Science:

• Medical Informatics
• Clinical Decision Support Systems
• Artificial Intelligence in Healthcare

Background:

• Venous thromboembolism (VTE) is a significant, preventable complication in hospitalized patients.
• Standardized VTE risk scoring using unstructured electronic health records (EHRs) is challenging.
• Current risk assessment tools require manual data extraction and interpretation.

Purpose of the Study:

• To evaluate the efficacy of large language models (LLMs) in automating VTE risk scoring.
• To compare expert knowledge-augmented prompts with basic and complex prompts for Padua and Caprini scores.
• To assess the performance of LLMs in risk stratification for thrombosis prevention.

Main Methods:

• Retrospective analysis of anonymized EHRs from a nationwide multicenter cohort (30 hospitals).
• Development of expert knowledge-augmented prompts for Padua and Caprini scores via a Delphi process.
• Evaluation of six open-source LLMs using prevalence-adjusted bias-adjusted kappa (PABAK) and F1 scores.

Main Results:

• Expert knowledge-augmented prompts achieved the highest performance for both Padua and Caprini scores.
• In the test dataset, mean item-level PABAK and F1 scores exceeded 0.90 for most items.
• Padua scoring demonstrated superior risk stratification (PABAK 0.92, F1 0.96) compared to Caprini (PABAK 0.73, F1 0.64).

Conclusions:

• LLMs with expert knowledge-augmented prompting can efficiently automate Padua and Caprini scoring.
• This approach supports automated risk stratification, enhancing thrombosis prevention workflows.
• The findings suggest a promising role for AI in improving VTE prophylaxis in clinical practice.