Comparing the effectiveness of generative AI technology in commonly asked scoliosis questions

In recent years, generative artificial intelligence systems have transformed the landscape of patient's access to medical information and education. As increases in general and subspeciality physician shortages lead to longer lead times for patients to get access to physicians, we aim to understand how effectively different AI platforms can respond to questions asked by parents about both operative and nonoperative scoliosis.

A survey comprised of 31 questions, among the most commonly asked, regarding scoliosis with responses from ChatGPT, Google Gemini, and Microsoft Copilot was administered to board-certified Orthopedic surgeons, fellowship trained in either pediatric or spine surgery. (four reviewers). They evaluated each output from Likert Scale of 1-5 with 5 meaning an excellent response was given and 1 meaning a poor response was given. Pairwise comparisons were used for analysis.

All three generative AI technologies performed well with an overall mean rating of 3.4 which is between good and very good on the Likert Scale provided. ChatGPT performed the best out of the three, with a mean rating of 4.0, Google Gemini was second best with a mean rating of 3.1, and Copilot was third best with a mean rating of 3.1. ChatGPT compared with Gemini and Copilot revealed statistically significant differences with a p-value <0.001, with no statistical difference between Gemini and Copilot.

In response to common scoliosis questions asked by parents, ChatGPT, Microsoft Copilot, and Google Gemini, were scored highly by our Spine team and has important indications for use in the future.