Diagnostic and Triage Performance of Large Language Models in Hand and Wrist Complaints: A Standardised Vignette...

Large language model (LLM) chatbots are widely accessible and increasingly used by the public for health information and as a tool to diagnose health conditions. While this may improve access to information, incorrect responses may lead to false reassurance and treatment delays. Evidence specific to 'patient-voice' descriptions in orthopaedic hand and wrist presentations is limited.

Fictitious first-person 'patient-voice' vignettes were created for common elective hand conditions (carpal tunnel syndrome, trigger finger, de Quervain tenosynovitis, thumb carpometacarpal osteoarthritis), common traumatic hand/wrist injuries (mallet finger, scaphoid, little finger metacarpal and distal radius fractures), and more minor injuries. Five vignettes were created per condition (20 elective; 20 trauma), and 10 minor-injury vignettes (total 50). Each vignette was entered into 3 freely available consumer LLMs (ChatGPT, Google Gemini, Microsoft Copilot). The first response was assessed for primary diagnostic and triage accuracy.

Overall diagnostic accuracy was 84.7%. Diagnostic accuracy differed by presentation type: 96.7% for elective, 65.0% for trauma, and 100% for minor injuries (P < .001). Overall triage accuracy was 83.3%, with 1.3% over-triage and 15.3% under-triage. Under-triage increased with acuity: 0.0% for minor injuries, 16.7% for elective presentations, and 21.7% for trauma (P = .0348).

Free-to-use LLM chatbots demonstrated high diagnostic and triage accuracy for minor injuries and elective hand conditions, but lower performance for traumatic presentations, with under-triage increasing with acuity. Given the potential for false reassurance and delayed care in time-critical injuries, LLM outputs should not be relied upon for triage decisions. Prospective evaluation using real-world patient queries is warranted.