Factors Associated With an LLM Arriving at the Same Diagnosis as a Musculoskeletal Specialist

Large language models (LLMs) can generate plausible diagnoses from patient symptom descriptions and convey complex medical information in conversational, empathetic language. Given that LLMs may struggle with the vague symptom descriptions characteristic of less healthy mindsets, discordance between LLM and clinician diagnoses might signal misinterpretation of sensations. Among new musculoskeletal outpatients, we studied factors associated with (1) diagnostic discordance between an LLM and a clinician and (2) patient rating of experience interacting with the LLM.

One hundred forty English-speaking patients described their symptoms to an LLM prompted to provide a single most likely diagnosis. Clinician diagnoses were recorded after the visit. Patients completed a survey assessing perceptions of the LLM interaction, demographics, and psychosocial factors-including measures of unhelpful thoughts and distress (eg, catastrophic thinking, misperception of pain as necessarily signifying injury, rumination about pain, and fear of losing cherished roles). Linear regression sought associations between personal factors, diagnostic concordance, and experience with the LLM.

Discordance between clinician and LLM diagnoses was common 45% (67 of 140), but was not associated with any factors. Discordance often reflected diagnostic ambiguity (eg, knee osteoarthritis and meniscal tear) or clinician use of specific diagnoses for nonspecific symptoms (eg, myofascial pain syndrome, complex regional pain syndrome, and piriformis syndrome). Hispanic ethnicity, unmarried status, lower educational attainment, and lower annual income were associated with more favorable patient-rated experience with the LLM.

Clinician use of speculative and ambiguous diagnostic labels may limit the usefulness of LLM-clinician discordance as a signal of patient unhelpful thinking and distress. LLMs may support personal health agency, particularly in the setting of social disadvantage.