Large language model-based identification of venous thromboembolism diagnostic delays

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Decision Support Systems

Background:

• Delayed diagnosis of venous thromboembolism (VTE) is a significant challenge in hospitalized patients.
• Current methods for identifying VTE diagnostic delays are limited, hindering quality improvement efforts.

Purpose of the Study:

• To develop and validate a novel large language model (LLM)-based electronic-trigger for detecting diagnostic delays in VTE.
• To assess the performance of the LLM-trigger compared to existing methods.

Main Methods:

• Utilized admissions data from internal medicine residents (n=20,843) over two years.
• Developed and validated LLM prompts to identify VTE consideration in notes and confirmation in imaging reports.
• Reviewed discordant cases to identify diagnostic delays, implementing exclusion criteria for accuracy.

Main Results:

• LLM prompts achieved high accuracy (98%-100%) in classifying medical notes and imaging studies.
• The LLM-trigger identified 137 potential diagnostic delays, with a positive predictive value (PPV) of 48% and negative predictive value (NPV) of 95% after exclusions.
• 4.4% of all VTE hospitalizations were identified as having a diagnostic delay.

Conclusions:

• The first LLM-based electronic-trigger for VTE diagnostic delays was successfully developed.
• This LLM-trigger demonstrates superior performance compared to non-LLM electronic-triggers.
• LLM applications show promise for scalable diagnostic feedback and quality improvement in healthcare.