Comparing computable structured phenotype- versus large language model-identification of opioid use disorder using...

Area of Science:

• Emergency Medicine
• Clinical Informatics
• Artificial Intelligence in Healthcare

Background:

• Opioid use disorder (OUD) is a significant public health issue, frequently encountered in emergency departments (EDs).
• Current identification methods, such as structured computable phenotypes, may not capture the full clinical context necessary for accurate diagnosis.

Purpose of the Study:

• To compare the performance of a structured computable OUD phenotype against a zero-shot large language model (LLM).
• To evaluate these methods against expert clinical review as the reference standard for OUD identification in the ED.

Main Methods:

• Retrospective analysis of 202 adult ED encounters.
• Expert physician review with consensus adjudication determined the reference standard for OUD status.
• A structured phenotype utilized ICD-10 codes, medications, toxicology, and keyword rules.
• A zero-shot LLM (GPT-4.1) classified OUD based on concatenated ED clinical notes.

Main Results:

• Expert reviewers achieved substantial agreement (κ=0.77), identifying OUD in 28% of encounters.
• The structured phenotype demonstrated high sensitivity (0.98) but low specificity (0.54).
• The zero-shot LLM achieved high sensitivity (0.93) and significantly improved specificity (0.90) compared to the phenotype (p<0.001).

Conclusions:

• A zero-shot LLM offers a balanced performance profile, excelling in specificity while maintaining high sensitivity for OUD detection.
• This suggests LLMs can be a valuable tool for tiered screening strategies in ED settings for opioid use disorder.
• LLM-based approaches may enhance the accuracy and efficiency of identifying OUD in emergency care.