Leveraging Large Language Models with Sequential Prompting to Extract Eye Examination Findings from Free-Text...

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Natural Language Processing

Background:

• Electronic health records (EHRs) contain valuable clinical information.
• Extracting specific findings from unstructured clinical notes is challenging.
• Automated methods are needed to efficiently process ophthalmology notes.

Purpose of the Study:

• To evaluate large language models (LLMs) for identifying slit lamp and fundus examination findings.
• To assess LLM performance in extracting eye examination data from free-text clinical notes.
• To compare LLM capabilities against established models like BERT.

Main Methods:

• A retrospective cohort study using ophthalmology progress notes from Stanford University.
• A novel sequential named entity recognition (NER) prompting approach was developed.
• Compared open-source LLMs (Mistral Nemo 12B, Llama 3.1 8B, Llama 3.1 70B) against fine-tuned BERT models.
• Tested on both templated (SmartLink-300) and unstructured (free-text-200) ophthalmology notes.

Main Results:

• Llama 3.1 70B achieved the highest performance, with a microaveraged F1 score of 0.92 on SmartLink notes.
• Llama 3.1 70B demonstrated strong performance across different eye examination components (FE: 0.94, SLE: 0.91).
• For unstructured notes, Llama 3.1 70B achieved high F1 scores (0.93-0.96) where BERT fine-tuning was not feasible.

Conclusions:

• Large language models with sequential prompting outperform or match fine-tuned BERT models for NER tasks.
• LLMs show flexibility and robustness on datasets unsuitable for BERT fine-tuning.
• LLMs offer a promising solution for automated, accurate information extraction from ophthalmology notes.