Dual perspectives on large language models in rheumatology: physician-rated quality and patient-centered usability of...

This study conducted an informatics system evaluation of two LLMs (GPT-4o and DeepSeek-V3) for patient education, combining clinician-rated quality with patient-perceived usability across thematically stratified queries.

In a blinded, within-subject design, 16 frequently asked questions about biologic therapies were categorized into three domains: treatment/drug selection, safety/adverse effects, and special conditions/daily life. Responses were standardized, generated without external retrieval, anonymized as A/B pairs. Thirty physicians assessed clinical appropriateness, scientific accuracy, comprehensiveness, while 60 patients rated readability, understandability, actionability, perceived adequacy, decision support, and trust on 5-point Likert scales. Analyses included paired t-tests, Holm/FDR corrections and two one-sided tests (TOST) to distinguish statistical non-difference from practical equivalence.

Physicians rated GPT higher across all domains (p < .002), with largest gaps in safety/side effects and treatment/drug selection. Patients favored GPT for understandability, actionability, and decision support (p < .001), while readability, adequacy, trust, and reading time were statistically and clinically equivalent.

Findings highlight the need for topic-aware governance: guideline-dense queries suited to retrieval-augmented generation and checklist compliance, and context-sensitive queries requiring uncertainty signaling and human oversight. This layered approach advances health informatics by defining where LLMs may substitute versus where they require verification, supporting safe and auditable integration into patient education.