Related Experiment Video
Updated: Jul 13, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Automated Review of Patient Records: Privacy-Preserving Large Language Models for Identifying Incident Nonarteritic
Tuyet Thao Nguyen1,2, Kelvin Zhenghao Li2,3,4, Pareena Chaitanuwong5,6
1University of California, Davis, Sacramento, California.
Purpose:
Retrospective identification of acute nonarteritic anterior ischemic optic neuropathy (NAION) cases is critical for research on risk factors. However, reliance on International Classification of Diseases (ICD) 10th edition coding for case identification has limited accuracy, and manual review of longitudinal electronic health records is time-intensive. The purpose of this study is to evaluate automated methods for retrospective identification of acute NAION cases using large language models (LLMs) that preserve patient privacy.
Design:
Retrospective cross-sectional study.
Subjects:
165 patients with ≥1 ICD-10 code for ischemic optic neuropathy (H47.01∗) in the electronic health record at an academic medical center.
Methods:
Five locally deployed LLM models (Mistral Small 3.1, Magistral Small, Gemma3, MedGemma, GPT-OSS 20B) were used to implement 4 approaches for acute NAION diagnostic classification using unstructured ophthalmology records (basic prompting, retrieval-augmented generation [RAG], 2-step agentic workflow, and 3-step agentic workflow). Ten percent of subjects were used for prompt refinement. Large language model/approach diagnostic classifications were compared against expert neuro-ophthalmologist diagnoses based on chart review.
Main Outcome Measures:
Positive predictive value (PPV) of LLM approaches for acute NAION case identification with expert chart review diagnosis serving as gold standard. Secondary outcomes included negative predictive value, sensitivity, specificity, accuracy, F1 score, and distribution of LLM/approach classifications.
Results:
7/17 prompt refinement subjects and 58/148 testing subjects had acute NAION by expert chart review corresponding to PPV of 0.39 for ≥1 ICD code. Large language model approaches accurately identified 20 ± 12 (mean, standard deviation) acute NAION cases in the test set with PPV of 0.78 ± 0.16 and accuracy of 0.69 ± 0.06. The Mistral model using a 3-step agentic approach had the best-balanced performance (39 cases identified, 0.85 PPV, 0.82 accuracy, 0.75 F1 score).
Conclusions:
Privacy-preserving agentic LLM approaches can achieve high PPV for acute NAION case identification using unstructured ophthalmology longitudinal electronic health records. These results exceed the performance of using structured ICD codes to identify cases, offering a scalable, efficient method for case identification in retrospective research while maintaining patient confidentiality and local data control. This method has application for enhancing research efficiency and accuracy for studies on NAION risk factors, with potential applicability to other conditions requiring complex diagnostic review.
Financial Disclosures:
Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.
