Extraction of Glaucoma Diagnosis, Type, and Severity from Clinical Notes using Secure Cloud-based Large Language...

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Health Informatics

Background:

• Electronic health records (EHRs) contain valuable clinical information in unstructured free text.
• Extracting specific diagnoses like glaucoma, its type, and severity from these notes is crucial for research and patient care.
• Current methods, such as ICD-10 coding, may not fully capture the nuances of clinical documentation.

Purpose of the Study:

• To assess the effectiveness of secure cloud-based large language models (LLMs) in extracting glaucoma diagnosis, type, and severity from EHR clinical notes.
• To compare LLM performance against expert clinician adjudication and traditional ICD-10 coding.

Main Methods:

• A retrospective chart review of 1,250 subjects from the Bascom Palmer Ophthalmic Repository was conducted.
• Clinical notes from 2014-2024 were annotated by glaucoma specialists for glaucoma presence, type, and severity.
• Five LLMs were evaluated via HIPAA-compliant containers on a held-out test set.
• Performance was measured using accuracy, sensitivity, specificity, and F1-scores, and compared to ICD-10 codes.

Main Results:

• High inter-grader agreement was observed for glaucoma detection, type, and severity.
• LLMs achieved high accuracy in glaucoma diagnosis (94.4%-97.5%) and type classification (94.0%-97.1%).
• LLMs also demonstrated strong performance in severity staging (94.0%-95.2%), significantly outperforming ICD-10 codes (58.5% accuracy).

Conclusions:

• Secure cloud-based LLMs effectively extract glaucoma information from clinical notes, nearing expert performance.
• LLMs significantly outperform ICD-10 coding for glaucoma phenotyping, especially for severity.
• LLMs show potential to convert unstructured EHR data into scalable, research-ready phenotypic data for ophthalmic research.