Development and Validation of a Large Language Model Case Identification Strategy for Eosinophilic Esophagitis

Area of Science:

• Medical Informatics
• Gastroenterology
• Computational Linguistics

Background:

• Epidemiologic research for eosinophilic esophagitis (EoE) faces challenges with current case identification algorithms.
• Accurate and efficient identification of EoE cases is crucial for understanding disease prevalence and characteristics.

Purpose of the Study:

• To evaluate rule-based natural language processing (RB-NLP) and large language model-based natural language processing (LLM-NLP) for identifying EoE diagnoses and features.
• To compare the performance of LLM-NLP and RB-NLP against International Classification of Diseases (ICD) codes for EoE case identification.

Main Methods:

• Gastrointestinal pathology reports and gastroenterology clinic notes mentioning "eosinophil" were analyzed.
• A dataset of 300 patients was divided into training (n=200) and testing (n=100) sets, with manual chart review as the reference standard.
• RB-NLP and LLM-NLP pipelines were developed and compared with ICD codes using sensitivity, positive predictive value, and F1 score.

Main Results:

• In the validation set, ICD codes alone achieved a sensitivity of 0.86 and an F1 score of 0.91.
• Combining ICD codes with LLM-assigned diagnoses showed a modest improvement in F1 score.
• The LLM + ICD approach identified more EoE cases in a larger cohort and captured 15% of cases missed by ICD codes alone.

Conclusions:

• Combining LLM-NLP with ICD codes reduces false negatives and modestly improves the F1 score for EoE case identification.
• This integrated approach offers a scalable method to enhance EoE case identification in real-world clinical data.
• LLM-NLP shows promise for improving the accuracy and efficiency of epidemiologic research in eosinophilic esophagitis.