Related Experiment Video
Updated: May 28, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Application of large language models in clinical decision-making for dry eye disease
Haoqiang Cui1, Yuyang Yang2, Taichen Lai2
1Department of Ophthalmology and Optometry, Fujian Medical University, Fuzhou, China.
Objective:
To systematically evaluate the diagnostic, classification, and treatment decision-making performance of large language models (LLMs) in dry eye disease (DED) and compare their performance with that of junior ophthalmologists to assess their feasibility as clinical support tools.
Methods:
One hundred standardized DED cases from Fuzhou University Affiliated Provincial Hospital (Aug 2023-Aug 2025) were analyzed. Four LLMs (ChatGPT-4o, DeepSeek-V3, Gemini-2.5-Pro, and ERNIE Bot-4.5-turbo) underwent an initial 20-item test with a 95% accuracy threshold. Only models meeting this criterion advanced to case-based evaluation. Diagnostic and therapeutic outputs were rated by senior ophthalmologists using the Global Quality Score (GQS) and a clinical safety score. Meanwhile, the response times of both the LLMs and the junior ophthalmologists were recorded to evaluate their respective efficiency.
Results:
ChatGPT-4o, DeepSeek-V3, and Gemini-2.5-Pro met the screening threshold. In determining treatment necessity, the accuracies of the LLMs (96-99%) were comparable to those of junior ophthalmologists (97%). For DED classification, DeepSeek-V3 achieved the highest accuracy (92%) and agreement with experts (kappa = 0.80), outperforming ChatGPT-4o and junior ophthalmologists (71%, kappa ≈ 0.53; p < 0.01). Gemini-2.5-Pro showed strong performance (accuracy 89%), the highest GQS (4.87 ± 0.37), and the best safety rating, with 89% of its outputs rated as "good". The decision efficiency of the LLMs was significantly higher (p ≤ 0.001), with response times (16-36 s) much shorter than those of junior ophthalmologists (315 ± 57 s).
Conclusions:
Gemini-2.5-Pro and DeepSeek-V3 demonstrated high accuracy, safety, and efficiency in case-based DED management, showing strong potential as auxiliary tools to enhance clinical decision-making and support less experienced clinicians.
Trial Registration:
Not applicable. This study is a retrospective observational study and does not involve any intervention requiring trial registration.