GLLaucoMed: A Secure Large Language Model-Powered Agentic Workflow for Automated Medication Extraction from Free-Text...

To evaluate the efficacy of large language models (LLMs) in extracting medication-related information from glaucoma clinical notes in the electronic health record.

Cross-sectional.

One thousand two hundred fifty subjects in the Bascom Palmer Ophthalmic Repository.

Extracted clinical notes from glaucoma-related encounters between 2014 and 2024 were randomly sampled and labeled by 2 glaucoma specialists, with a third serving as an adjudicator. Graders were asked to label current topical medications (CTM), proposed changes to topical medications (ΔTM), current oral medications (COM), and proposed changes to oral medications (ΔOM) in a structured fashion. The data set was split into development (10%), validation (10%), and test (80%) sets stratified by clinician. Development and validation sets were used to engineer and refine prompts, and the held-out test set was used for model assessment. Five LLMs (Claude Opus 4.6, DeepSeek-V3.2, GPT 5.2, Grok 4.1, and Qwen3.6-35B-A3B) were accessed via Microsoft Azure AI Foundry within a Health Insurance Portability and Accountability Act-compliant environment. Intergrader agreement was assessed with Gwet AC1. Large language model performance was initially assessed in a binary fashion with F1 scores, and the degree of text match among positive cases was evaluated using exact match accuracy and Jaccard Index (JI).

F1 score, exact match accuracy, JI.

Gwet AC1 for intergrader agreement was 0.799, 0.888, 0.985, and 0.988 for CTM, ΔTM, COM, and ΔOM, respectively. F1 scores for CTM were 0.985, 0.971, 0.978, 0.968, and 0.970 for Claude, DeepSeek, GPT, Grok, and Qwen, respectively; for ΔTM: 0.905, 0.826, 0.897, 0.842, and 0.865, respectively; for COM: 0.923, 0.887, 0.899, 0.906, and 0.894, respectively; for ΔOM: 0.958, 0.815, 0.937, 0.835, and 0.940, respectively. Among positive cases, range of exact match accuracies for CTM (n = 1354) was 0.730-0.882, and range of JIs was 0.809-0.918. For ΔTM (n = 404), exact match accuracy range was 0.619 to 0.780, and JI range was 0.668-0.827. For COM (n = 47), exact match accuracy range was 0.766 to 0.872, and JI range was 0.765 to 0.870. For ΔOM (n = 25), exact match accuracy range was 0.583 to 0.920, and JI range was 0.583 to 0.922.

The GLLaucoMed pipeline demonstrated high performance in extracting and standardizing medication data from unstructured clinical notes, including both current medications and proposed changes. Claude and GPT exhibited the strongest performance.

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.