Comparative analysis of large language models and clinicians in thyroid eye disease using structured questionnaires

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Medical Informatics

Background:

• Thyroid eye disease (TED) management requires complex clinical decision-making.
• Large language models (LLMs) are increasingly explored for medical applications.
• Evaluating LLM agreement with expert clinicians is crucial for their safe integration.

Purpose of the Study:

• To assess the preliminary agreement between LLMs and ophthalmic clinicians in TED clinical decision-making.
• To compare the performance of GPT-5 and Gemini 2.5 Pro in a TED scenario questionnaire.
• To establish a baseline for future research on AI in ophthalmology.

Main Methods:

• A structured online survey with TED-related scenarios was administered to 17 oculoplastic/orbital clinicians.
• The same questionnaire was posed to GPT-5 and Gemini 2.5 Pro.
• Agreement was measured using Jaccard index and cosine similarity, with inter-physician agreement as a baseline.

Main Results:

• LLMs demonstrated moderate overall agreement with clinician responses.
• GPT-5 showed slightly higher alignment than Gemini 2.5 Pro in demographic and diagnostic domains (p<0.05).
• No significant difference in agreement was found based on clinician experience levels.

Conclusions:

• LLMs exhibit partial alignment with clinician decision-making patterns in structured, decontextualized settings.
• Current LLM performance in TED decision-making requires further investigation with real-world data.
• Future studies should incorporate open-ended clinical tasks and patient data for comprehensive evaluation.