Zero-shot LLM-based visual acuity extraction: a pilot study

Visual acuity (VA) and intraocular pressure (IOP) measurements are crucial indicators of ocular health. However, the documentation of these vital parameters in clinical notes often lacks standardisation. The aim of this study was to evaluate the feasibility of using of large language models (LLMs) for automated extraction of VA and IOP data from unstructured ophthalmology clinic notes.

Outpatient clinic notes from The Queen Elizabeth Hospital Ophthalmology department were analysed using a 70 billion parameter LLaMA-3 model in a zero-shot learning approach. Nine data points per eye were extracted, including various VA measurements, IOP, and measurement methods. Results were compared to expert-verified ground truth data.

Sixty-nine outpatient clinic notes were collected. Locally deployed LLM analysis of clinic notes was feasible. High accuracy was observed in extracting best corrected VA and IOP (above 90% for both eyes). Performance varied for other measurements, with lower accuracy for uncorrected VA (67-75%) and challenges in interpreting ambiguous documentation. The model struggled to identify assumed uncorrected VA cases, highlighting issues which may correlate with documentation clarity.

This study has demonstrated that it is feasible to deploy a LLM locally to extract VA data from free-text notes. However, the accuracy of extracted data was lacking in several domains. These performance issues highlight that further research is required to optimise the accuracy of the retrieved data.