Development and Validation of a Large Language Model Case Identification Strategy for Eosinophilic Esophagitis

Epidemiologic research in eosinophilic esophagitis (EoE) is limited by the accuracy and efficiency of case identification algorithms. We aimed to evaluate rule-based natural language processing (RB-NLP) and large language model-based natural language processing (LLM-NLP) pipelines for identifying EoE diagnoses and features from unstructured text.

We identified gastrointestinal pathology reports with any mention of "eosinophil" paired with gastroenterology clinic notes. Three hundred randomly selected patients were divided into training (n = 200, 56 with EoE) and testing (n = 100, 36 with EoE) sets. Manual chart review was the reference standard. RB-NLP used spaCy with medspaCy's clinical components; LLM-NLP prompts were developed through iterative human-in-the-loop refinement. In the validation set, we compared International Classification of Diseases (ICD) codes, RB-NLP, and LLM-NLP against the reference standard using sensitivity (recall), positive predictive value (precision), and F1 score.

In the validation set, ICD codes alone had a sensitivity 0.86 (95% confidence interval [CI]: 0.75-0.97), a positive predictive value of 0.97 (95% CI: 0.91-1.0), and an F1 value of 0.91 (95% CI: 0.84-1.0). Combining ICD and LLM-assigned diagnosis yielded a 3-point improvement in F1 score (95% CI: -0.01 to 0.07; P = .2) compared to ICD alone. In a larger cohort (n = 580), the LLM + ICD approach identified the most EoE cases (n = 203) and captured 15% of cases missed by ICD codes. Clinical characteristics varied depending on the case identification strategy used.

Combining LLM-NLP with a single ICD code reduced false negatives and modestly improved the F1 score compared to either method alone. This may represent a scalable approach to enhance EoE case identification in real-world data.