Zero-shot LLM-based visual acuity extraction: a pilot study

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Medical Informatics

Background:

• Visual acuity (VA) and intraocular pressure (IOP) are key indicators of eye health.
• Standardized documentation of VA and IOP in clinical notes is often lacking.
• Automated data extraction from unstructured clinical notes is needed.

Purpose of the Study:

• To assess the feasibility of using large language models (LLMs) for automated extraction of VA and IOP data.
• To evaluate the accuracy of LLM-extracted data compared to expert-verified records.

Main Methods:

• Analysis of 69 ophthalmology outpatient clinic notes using a 70 billion parameter LLaMA-3 model.
• Zero-shot learning approach for data extraction, targeting nine data points per eye.
• Comparison of LLM-extracted data against expert-verified ground truth.

Main Results:

• Local LLM deployment for clinic note analysis was feasible.
• High accuracy (>90%) achieved for best-corrected VA and IOP extraction.
• Lower accuracy (67-75%) for uncorrected VA, with challenges in ambiguous documentation.

Conclusions:

• LLMs can be locally deployed for extracting VA data from free-text ophthalmology notes.
• Current LLM accuracy is insufficient in several data domains, particularly for uncorrected VA.
• Further research is necessary to enhance the precision of LLM-based data retrieval from clinical records.