Computable Structured Phenotype Versus Large Language Model Identification of Opioid Use Disorder Using Electronic...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Decision Support

Background:

• Accurate identification of opioid use disorder (OUD) in the emergency department (ED) is crucial for timely intervention.
• Existing rule-based computable phenotypes offer a structured approach to OUD detection.
• The emergence of large language models (LLMs) presents new opportunities for clinical data analysis.

Purpose of the Study:

• To compare the diagnostic performance of a rule-based computable phenotype against a large language model (LLM) for identifying OUD in ED encounters.
• To utilize expert physician review as the gold standard for OUD diagnosis.

Main Methods:

• Retrospective analysis of adult ED encounters with random sampling.
• Development of a rule-based phenotype incorporating diagnosis codes, medications, toxicology, and keywords.
• Application of a large language model (ChatGPT 4.1) using zero-shot prompting on ED notes.
• Independent chart review by two emergency physicians for OUD determination, with adjudication for discrepancies.
• Estimation of performance metrics (sensitivity, specificity, PPV, NPV) using inverse probability weighting.

Main Results:

• Weighted prevalence of OUD was 5.6% in the study population.
• The rule-based phenotype achieved a sensitivity of 0.84 and specificity of 0.964.
• The large language model demonstrated a sensitivity of 0.81 and a significantly higher specificity of 0.996 (P<.0001).
• The LLM also showed a higher positive predictive value (0.92) compared to the phenotype (0.58).

Conclusions:

• Both the rule-based phenotype and the large language model exhibit strong diagnostic capabilities for OUD in the ED.
• The LLM's superior specificity and positive predictive value suggest its potential to minimize false-positive OUD identifications in clinical workflows.
• Further prospective validation of LLMs in diverse patient populations is warranted.