Artificial intelligence versus orthopedic surgeons as an orthopedic consultant in the emergency department

ChatGPT, a widely accessible AI program, has demonstrated potential in various healthcare applications, including emergency department (ED) triage, differential diagnosis, and patient education. However, its potential in providing recommendations to emergency department providers with orthopedic consultations has not been evaluated yet.

This study compared the performance of four board certified orthopedic surgeons, two attendings and two trauma fellows who take independent call at the same institution and ChatGPT-4 in responding to clinical scenarios commonly encountered in emergency departments. Five common orthopedic ED scenarios were developed (lateral malleolar ankle fractures, distal radius fractures, septic arthritis of the knee, shoulder dislocations, and Achilles tendon ruptures), each with four questions related to diagnosis, management, surgical indication, and patient counseling, totaling 20 questions. Responses were anonymized, coded, and evaluated by independent reviewers including emergency medicine physicians using a five-point Likert scale across five criteria: accuracy, completeness, helpfulness, specificity, and overall quality.

When comparing the ratings of AI answers to non-AI responders, the AI answers were shown to be superior in completeness, helpfulness, specificity, and overall quality with no difference in regards to accuracy (p < 0.05). When considering question subtypes including diagnosis, management, treatment, and patient counseling, AI was shown to have superior scores in helpfulness, and specificity in diagnostic questions(p < 0.05). In addition, AI responses were superior in all the assessed categories when looking at the patient counseling questions (p < 0.05). When considering different clinical scenarios, AI outperformed non-AI groups in completeness in the distal radius fracture scenario. Furthermore, AI outperformed non-AI groups in helpfulness in the lateral malleolus fracture scenario. In the shoulder dislocation scenario, AI responses were more complete, helpful, and had a better overall quality. AI responses were non-inferior in the remaining categories of the different scenarios.

Artificial intelligence exhibited non-inferior and often superior performance in common orthopedic-ED consultations compared to board certified orthopedic surgeons While current AI models are limited in their ability to integrate specific images and patient scenarios, our findings suggest AI can provide high quality recommendations for generic orthopedic consultations and with further development, will likely have an increasing role in the future.