Can natural language processing serve as a consultant in oral surgery?

In this comprehensive evaluation, ten experienced oral surgeon experts posed a total of twenty oral surgery-related questions, including dental implant and tooth extractions, to three distinct Natural Language Processing (NLP)-based chatbot platforms: ChatGPT, Microsoft Bing, and Google Bard. The study aimed to assess the effectiveness of these chatbots in responding to specialized medical questions.

Two primary evaluation metrics were employed: a Likert Scale (LS) for measuring the accuracy and completeness of responses and a Global Quality Scale (GQS) for evaluating the clarity of responses. Statistical analyses, including one-way analysis of variance (ANOVA) and Post Hoc Tukey, were conducted to assess and compare the performance of the chatbots as rated by the experts.

The results of the study revealed significant differences in the performance of the chatbots. ChatGPT statistically achieved a better mean LS score of 1.4000±0.15986 than Microsoft Bing (1.8750±0.18143) and Google Bards (2.0500±0.12472) (P < 0.001). Additionally, ChatGPT statistically achieved a higher GQS score of 4.4200±0.30111 than Microsoft Bing (3.7550±0.28621) and Google Bards (3.5250±0.22392) (P < 0.001).

These findings showed the substantial advantage of ChatGPT in effectively addressing oral surgery-related questions with superior accuracy, completeness, and clarity. The study highlights the potential of advanced NLP platforms to enhance information retrieval and communication within the field of oral surgery, reinforcing the utility of such technologies in medical and surgical domains.