Identification of primary sclerosing cholangitis: ICD-10 code validation and comparison with a large language model...

Area of Science:

• Hepatology and Gastroenterology
• Medical Informatics
• Clinical Epidemiology

Background:

• Retrospective studies on primary sclerosing cholangitis (PSC) lacked a specific diagnostic code, hindering research.
• A dedicated PSC-specific ICD-10 code was introduced in 2018.

Purpose of the Study:

• To validate the new PSC-specific ICD-10 code for patient identification.
• To compare the ICD-10 code's efficacy against other methods, including natural language processing (NLP) and a prediction model.

Main Methods:

• Utilized NLP to extract potential PSC cases from electronic health records.
• Physician adjudication confirmed PSC diagnoses.
• Developed and validated a PSC prediction model using LASSO regression.
• Assessed the diagnostic performance of the ICD-10 code, LASSO model, and large language models (LLMs).

Main Results:

• The PSC-specific ICD-10 code demonstrated strong discriminatory capacity (c-statistic=0.87).
• A LASSO-based prediction model showed slightly improved performance (c-statistic=0.92).
• LLMs exhibited inferior discrimination (c-statistic=0.77) compared to the ICD-10 code and LASSO model.

Conclusions:

• The PSC-specific ICD-10 code is a reliable tool for identifying PSC patients in database studies.
• While prediction models offer marginal improvements, the ICD-10 code alone is adequate for research purposes.
• LLMs were less effective than the ICD-10 code or prediction models for PSC identification.