A patient-derived benchmark for evaluating large language models in connective tissue diseases: blinded...

To co-develop disease-specific patient questions for connective tissue diseases (CTDs), compare patient/rheumatologist ratings of answers from language models (LLMs) versus Google Search, and quantify EULAR coverage of these patient-prioritized questions. In this prospective single-center observational study, reported in accordance with STROBE, patient advocacy groups curated 20 frequently asked questions (FAQs) for each CTD (systemic lupus erythematosus, idiopathic inflammatory myopathy, Sjögren disease, systemic sclerosis). Questions were submitted to Claude 4.0 Sonnet, ChatGPT-5, Gemini 2.5 Pro, and Google Search. Five patients per disease and five rheumatologists rated blinded outputs using forced-rank preference and 5-point Likert scales (patients: empathy, trustworthiness, comprehensibility; physicians: medical correctness, empathy, comprehensibility). Guideline mapping assessed whether EULAR recommendations addressed each question. All three LLMs answered 100% of questions (80/80), whereas Google Search answered 90% (72/80). Across CTDs, patients rated LLM outputs favorably for empathy (mean 1.40-2.77), trustworthiness (1.47-2.6), and comprehensibility (1.33-2.27), and rheumatologists for medical correctness (1.23-1.98). Patients and physicians most often ranked Gemini 2.5 Pro best overall (rank 1: 59% and 63%), and Google Search most often worst (rank 4: 55% and 57%). Cluster analyses showed no consistent differences across five clusters. Guideline mapping revealed substantial gaps: 40-55% of prioritized FAQs were not addressed by EULAR recommendations, with largest gaps in cluster E, life impact, psychosocial aspects and family planning (75-100% not addressed; 0% fully addressed). In this blinded evaluation, LLMs produced CTD FAQs responses patients perceived as empathic, trustworthy and rheumatologists rated medically correct, with Gemini 2.5 Pro most consistently preferred overall. However, the mismatch between prioritized questions and guideline coverage underscores the need for patient-centered, evidence-grounded information resources.