Assessment of the information quality of chatbot technologies on orthodontic miniscrews

In this era of artificial intelligence, the increasing competition has significantly enhanced the power of AI chatbots, which have been integrated into various fields, including orthodontics. They can be a good option for enlightening topics of curiosity before or during treatment, as they are easily accessible by patients. However, their reliability remains a concern. Miniscrews, or Temporary Anchorage Devices (TADs), frequently used during orthodontic treatments, are among the subjects of interest to patients.

This study aimed to comparatively evaluate the responses provided by four Large Language Models (LLMs), namely ChatGPT-3.5, ChatGPT-4 (OpenAI), Google Bard (Google LLC), and Bing Chat (Microsoft Corp), to questions asked by patients in the field of orthodontic miniscrews.

The most frequently asked questions by patients about miniscrews used in orthodontics were searched on Google. The first 50 pages were reviewed, and 30 questions were selected and asked to the LLM. The responses from the LLMs were evaluated using a five point modified scale and modified DISCERN (mDISCERN) by three orthodontic residents.

It was observed that the highest score for Likert belonged to GPT-4 (3.84), while the lowest was for Bing Chat (3.37). Statistically significant differences were found between the median score values given to the questions by all three researchers, depending on the LLMs used (p = 0.016; p<0.001 and 0.017, respectively). Significant differences were found between the scores given by Investigator 1 and Investigator 3 to ChatGPT-4 and Google Bard and the scores given to Bing Chat. Additionally, Investigator 2 showed significant differences in the scores given to Bing Chat compared to ChatGPT-3.5 (p=0.018). Among the evaluated chatbots, ChatGPT-4 achieved the highest mDISCERN score (23.43 ± 2.89), followed by Google Bard (23.47 ± 3.2), ChatGPT-3.5 (22.62 ± 2.97), and Bing Chat (21.63 ± 2.43).

In this study, it was found that LLMs can generally inform patients about miniscrews used in orthodontics and have promising potential. However, it is necessary that the information provided by these programs should always be supported by the information given by professionals.