Related Experiment Videos
Artificial Intelligence Can Direct Patients Toward a Complaint-specific Musculoskeletal Provider
Ethan C Gazan1, Colin M Emrich, Alexander J Baur
1From the Liberty University College of Osteopathic Medicine (Gazan, Emrich, Dr. Baur, Dr. Landy), Lynchburg, VA; the Department of Orthopaedic Surgery (Dr. Lum), University of California Davis, Sacramento, CA; the Connecticut Orthopaedics (Dr. Bernstein), Trumbull, CT, and the OrthoVirginia (Dr. Landy), Lynchburg, VA.
Introduction:
Patients with musculoskeletal complaints often search online to identify an appropriate healthcare provider. With the increasing availability of large language models (LLMs), these artificial intelligence (AI) tools can direct patients to providers. This study evaluated the ability of LLMs to recommend appropriate providers based on representative patient musculoskeletal queries.
Methods:
Three LLMs (ChatGPT, DeepSeek, and Gemini) were prompted with standardized musculoskeletal queries for two US cities (Lynchburg, VA, and Trumbull, CT). Provider recommendations were considered appropriate if the physician was currently practicing in the requested location and specialized in the relevant area. Listed phone numbers were checked for accuracy. Descriptive statistics and Fisher exact tests were used to summarize findings.
Results:
The appropriateness of recommended providers differed across models with ChatGPT being most often appropriate (17/17, 100%) compared with Gemini (9/21, 43%) and DeepSeek (4/10, 40%), (P < 0.001). Of the 18 inappropriate recommendations, 13 (72%) were real providers in unrelated specialties and 5 (28%) were hallucinations, all from DeepSeek. Phone number accuracy differed significantly across models with Gemini being most accurate (5/6, 83%), outperforming both ChatGPT (6/9, 67%; P = 0.60) and DeepSeek (2/10, 20%; P = 0.04).
Discussion:
LLMs showed potential to direct patients to local, specialized musculoskeletal providers based on their report, although the specific contact information was at times inaccurate. As these tools evolve, providers should be aware of AI's ability to make provider recommendations and work to ensure the presentation of their contact information is accessible by these models as best possible.
