A Blinded Comparison of Three Generative Artificial Intelligence Chatbots for Orthopaedic Surgery Therapeutic...

Area of Science:

• Artificial Intelligence in Medicine
• Orthopaedic Surgery
• Medical Information Systems

Background:

• Chatbots offer potential for accessing medical information.
• Evaluating the accuracy and reliability of chatbot responses in specialized fields like orthopaedic surgery is crucial.
• Existing chatbots (ChatGPT, Bing Chat) and a specialized tool (AskOE) were assessed.

Purpose of the Study:

• To compare the quality of responses from ChatGPT, Bing Chat, and AskOE for orthopaedic surgery therapeutic treatment questions.
• To identify which chatbot provides the most clinically correct, complete, safe, and useful information.

Main Methods:

• Orthopaedic surgery experts from McMaster University and the University of Toronto evaluated chatbot responses.
• A standardized rubric assessed clinical correctness, completeness, safety, usefulness, and references.
• Mixed effects logistic regression analyzed factors influencing chatbot preference.

Main Results:

• AskOE was significantly preferred over ChatGPT and Bing Chat (P<0.001) by reviewers.
• AskOE received significantly higher total evaluation scores compared to ChatGPT and Bing Chat (P<0.001).
• Clinical correctness, completeness, usefulness, and references were key factors associated with preferring AskOE.

Conclusions:

• AskOE demonstrated superior performance in providing accurate and reliable orthopaedic surgery treatment information compared to general chatbots.
• The findings suggest AskOE has significant potential as a trustworthy resource in healthcare settings for orthopaedic surgery.
• Further research into AI applications for specialized medical fields is warranted.