Acute Achilles tendon rupture: how well can artificial intelligence chatbots answer patient inquiries?

Artificial intelligence (AI) chatbots have gained popularity as a source of information that is easily accessed by patients. The best treatment of acute Achilles tendon ruptures (AATR) remains controversial due to varying surgical repair techniques, postoperative protocols, nonoperative treatment options, and surgeon and patient factors. Given that patients will continue to turn towards AI for answers to medical questions, the purpose of this study is to evaluate whether popular AI engines can provide adequate responses to frequently asked questions regarding AATR.

Three AI engines (ChatGPT, Google Gemini, and Microsoft Copilot) were prompted for a concise response to ten common questions regarding AATR management. Four board-certified orthopaedic surgeons were asked to assess the responses using a four-point scale. A Kruskal-Wallis test was used to compare the responses between the three AI systems using the scores assigned by the surgeons.

All three engines provided comparable answers to 7 of 10 questions (70%). Significant differences were noted between the AI systems for three of the ten questions (Question 4, overall p = .027; Question 7, overall p = .043; and Question 10, overall p = .033). post-hoc analyses revealed that Copilot received significantly poorer scores (higher mean ratings) compared to Gemini for Question 4 (adjusted p = .028) and Question 7 (adjusted p = .036), and poorer score compared to ChatGPT for Question 10 (adjusted p = .033).

AI chatbots can appropriately answer concise prompts about diagnosis and management of AATR. The responses provided by the three AI chatbots analyzed in our study were largely uniform and satisfactory, with only one of the engines scoring lower on three of the ten questions. As AI engines advance, they will become an important tool for patient education in orthopaedics.