Related Experiment Video
Updated: May 31, 2025
Author Spotlight: Advancing CBCT and Digital Dental Image Integration with AI-Assisted Digitization
Published on: February 23, 2024
Accuracy of Artificial Intelligence Based Chatbots in Analyzing Orthopedic Pathologies: An Experimental
Tobias Gehlen1,2, Theresa Joost3, Philipp Solbrig1
1Center for Musculoskeletal Surgery, Charité-Universitätsmedizin Berlin, Corporate Member of Freie Universität Berlin, Humboldt-Universität zu Berlin, and Berlin Institute of Health, 13353 Berlin, Germany.
Artificial intelligence (AI) symptom checker apps do not yet outperform physicians in diagnosing orthopedic and trauma cases or assessing urgency. While AI shows potential, current diagnostic accuracy and urgency assessment by these apps are significantly lower than human experts.
Area of Science:
- Medical Artificial Intelligence
- Digital Health
- Orthopedics and Traumatology
Background:
- Artificial intelligence (AI) is rapidly advancing, offering new diagnostic possibilities in medicine.
- Symptom checker applications (apps) show promise for patient decision support.
- The comparative diagnostic accuracy and urgency assessment of AI symptom checkers versus physicians remain largely unexamined.
Purpose of the Study:
- To evaluate the performance of current AI symptom checker apps in orthopedic and traumatology.
- To compare the diagnostic accuracy and urgency assessment of these apps against human orthopedic and trauma specialists.
Main Methods:
- Thirty fictitious orthopedic and trauma case vignettes were analyzed.
- Four orthopedic specialists and four distinct AI symptom checker apps evaluated each case for diagnostic accuracy and urgency.
- Statistical analysis, including Spearman's correlation, was used to compare physician and app performance (p < 0.05).
Main Results:
- Physicians achieved a correct diagnosis in 84.4% of cases, while AI apps achieved 35.8%.
- Physicians correctly assessed urgency in 70.0% of cases, compared to 20.6% for AI apps.
- Diagnostic accuracy agreement varied (low to high) between physicians and apps; urgency assessment agreement was moderate to high.
Conclusions:
- AI-based symptom checker apps currently lag behind physicians in diagnostic accuracy and urgency evaluation for orthopedic and trauma conditions.
- Significant variability exists in the accuracy of different AI digital tools.
- The field of AI in medical diagnosis shows considerable potential and warrants further investigation.
