Risk-centered benchmarking of large language models for AI-enabled counseling in chronic autoimmune thyroid eye...

Area of Science:

• Ophthalmology
• Endocrinology
• Artificial Intelligence in Healthcare

Background:

• Thyroid eye disease (TED) is a chronic autoimmune inflammatory orbital condition requiring careful management.
• Patients increasingly use large language models (LLMs) for health information, yet their quality for TED counseling is under-researched.

Purpose of the Study:

• To benchmark the quality and safety of web-based LLMs for thyroid eye disease (TED) counseling.
• To evaluate LLM performance across key aspects of patient information delivery.

Main Methods:

• A cross-sectional study utilized a 35-question Chinese TED counseling benchmark covering symptoms, treatment, and care-seeking.
• Five LLMs (Gemini 3 Pro, ChatGPT-5.2, DeepSeek-V3.1, Doubao, Qwen3-Max) were evaluated via single-turn prompts.
• Expert raters assessed Accuracy, Logic, Coherence, Safety, and Content Accessibility using Likert scales; response times and output features were analyzed.

Main Results:

• Significant differences were observed in response times and output characteristics (length, formatting) across LLMs.
• LLMs varied significantly in Accuracy, Logic, Coherence, and Content Accessibility, but not in Safety.
• Response time correlated with output length, but longer or slower responses did not consistently correlate with higher clinical quality.

Conclusions:

• There is marked heterogeneity in LLM efficiency, structure, and clinical quality for TED counseling.
• Current LLM safety performance should not be assumed as equivalent or absolute.
• Further validation of LLMs, focusing on structured, risk-centered outputs and multi-turn dialogues, is warranted.