Leveraging Large Language Models with Sequential Prompting to Extract Eye Examination Findings from Free-Text...

The purpose of this study was to evaluate the ability of general domain large language models (LLMs) to identify slit lamp and fundus examination (FE) findings from free-text clinical notes in electronic health records.

Retrospective cohort study.

Data were obtained from the Stanford Research Repository, which contains progress notes of patients from the Department of Ophthalmology at Stanford University since 2008.

We assessed a local LLM pipeline, using a novel sequential named entity recognition (NER) prompting approach, for identifying eye examination findings and their lateralities from free-text clinical notes at a single academic center. We compared several local open-source LLMs, including Mistral Nemo 12B, Llama 3.1 8B, and a quantized Llama 3.1 70B. Results were compared to those from a prior study that evaluated Bidirectional Encoder Representations from Transformers (BERT) models fine-tuned on 31 279 weakly labeled ophthalmology progress notes. Testing was conducted on 2 sets: (1) SmartLink-300 templated ophthalmology notes (>5900 entities) and (2) free-text-200 unstructured notes (>3000 entities) from 3 ophthalmologists.

Precision, recall, and F1 score for each component in the SLE and FE.

Llama 3.1 70B was the top-performing model for identifying eye examination findings from SmartLink notes, achieving a microaveraged precision of 0.95, recall of 0.90, and F1 score of 0.92. Llama 3.1 70B performed consistently well across different components of the eye examination, with a microaveraged F1 score of 0.94 for the FE and 0.91 for SLE. For the free-text notes, where fine-tuning a BERT model was not possible, Llama 3.1 70B achieved microaveraged F1 scores of 0.93 with ophthalmologist A notes, 0.94 with ophthalmologist B notes, and 0.96 with ophthalmologist C notes.

Sufficiently large LLMs, with our sequential prompting approach, outperformed or performed comparably to BERT models fine-tuned on >30 000 examples on the NER task and demonstrated flexibility by performing robustly on data sets BERT could not be fine-tuned on. Our findings suggest LLMs are capable of efficient, automated, and accurate extraction of relevant information from ophthalmology free-text progress notes.

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.