Comparing AAOS appropriate use criteria with ChatGPT-4o recommendations on treating distal radius fractures

Area of Science:

• Orthopedic Surgery
• Artificial Intelligence in Medicine
• Clinical Decision Support Systems

Background:

• The American Academy of Orthopaedic Surgeons (AAOS) established Appropriate Use Criteria (AUC) for distal radius fracture management.
• Evaluating the accuracy of advanced AI models like ChatGPT-4o against established clinical guidelines is crucial.

Purpose of the Study:

• To assess the accuracy of ChatGPT-4o's treatment appropriateness scores for distal radius fractures.
• To compare ChatGPT-4o's recommendations against the AAOS AUC for distal radius fracture management.

Main Methods:

• 240 distal radius fracture scenarios were evaluated using AO/OTA classification, injury mechanism, patient activity, health status, and associated injuries.
• Treatment appropriateness was scored by orthopedic surgeons (AAOS AUC) and ChatGPT-4o.
• Statistical analysis included error metrics (mean error, MAE, MSE) and Spearman correlation to compare scores.

Main Results:

• Significant positive correlations were found for dorsal spanning bridge (0.43) and spanning external fixation (0.4) between AAOS AUC and ChatGPT-4o.
• High agreement (99.17%) was observed for immobilization without reduction, and 90.42% for volar locking plates.
• Lower agreement (15%) was noted for dorsal plating, indicating variable concordance across treatment options.

Conclusions:

• ChatGPT-4o demonstrates inconsistent alignment with AAOS AUC for distal radius fracture treatment.
• The AI model's tendency towards conservative management raises concerns regarding its reliability for clinical decision support.
• Further validation is needed before integrating AI recommendations into orthopedic treatment planning.