Related Experiment Video
Updated: Sep 4, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Patient education for neuromyelitis optica spectrum disorder using large language models: combining expert assessment
Chen Li1, Yuting Hu2, Xiaoyan Wang2
1Department of Ophthalmology, The First Affiliated Hospital of Soochow University, Suzhou, China.
Objectives:
This study aimed to compare the educational performance of six mainstream LLMs for neuromyelitis optica spectrum disorder (NMOSD) and evaluated patient satisfaction during real-world interactions.
Methods:
This study was conducted from March to April 2026. In the first Phase, Twenty NMOSD-related questions derived from clinical guidelines and patient concerns were submitted to six LLMs (ChatGPT-5.4, Gemini-3.1-pro, Claude-4.6-Sonnet, DeepSeek-3.2, Kimi-2.5, and Qwen-3.5-plus). Responses were anonymized and independently evaluated by three neuro-ophthalmology specialists using Likert framework assessing accuracy, completeness, readability, safety, and humanity. Inter-rater reliability was assessed using the intraclass correlation coefficient (ICC). In the second Phase, the three best-performing models were subsequently evaluated through real-world interactions with ten NMOSD patients, and satisfaction scores were analyzed using linear mixed-effects models.
Results:
A total of 120 chatbot responses were evaluated. With a comprehensive evaluation, significant differences were observed across all assessment domains. Gemini-3.1-pro achieved the highest scores for accuracy and safety, while Qwen-3.5-plus demonstrated superior completeness and humanity. DeepSeek-3.2 generated the most accessible responses, exhibiting the lowest reading difficulty score. However, its completeness advantage should be interpreted with caution, as it may be partially influenced by its longer response length. Inter-rater reliability was good, with single-measure ICC values ranging from 0.535 to 0.759, and average-measure ICC values ranging from 0.775 to 0.904. In patient interactions, Qwen-3.5-plus achieved the highest satisfaction score, significantly outperforming Gemini-3.1-pro and DeepSeek-3.2. Although all LLMs demonstrate superior performance in patient education, the real-world interaction with patient needs to pay attention.
Conclusions:
LLMs demonstrate considerable potential for NMOSD patient education but exhibit variability across educational dimensions, and require further validation in larger cohorts. These findings highlight the importance of selecting LLMs according to specific patient education goals and underscore the importance of clinicians in rare disease counseling.
