Artificial Intelligence Chatbots as Sources of Implant Dentistry Information for the Public: Validity and Reliability...

This study assessed the reliability and validity of responses from three chatbot systems-OpenAI's GPT-3.5, Gemini, and Copilot-concerning frequently asked questions (FAQs) in implant dentistry posed by patients.

Twenty FAQs were prompted to three chatbots in three different times utilizing their respective application programming interfaces. The responses were assessed for validity (low and high threshold) and reliability by two prosthodontic consultants using a five-point Likert scale.

The test of normality was utilized using the Shapiro-Wilk test. Differences between different chatbots regarding the quantitative variables in a given (fixed) time point and between the same chatbots in different time points were assessed using Friedman's two-way analysis of variance by ranks, followed by pairwise comparisons. All statistical analyses were conducted using the SPSS (Statistical Package for Social Sciences) Version 26.0 software program.

GPT-3.5 provided the longest responses, while Gemini was the most concise. All chatbots advised consulting dental professionals more frequently. Validity was high under the low-threshold test but low under the high-threshold test, with Copilot scoring the highest. Reliability was high for all, with Gemini achieving perfect consistency.

Chatbots showed consistent and generally valid responses with some variability in accuracy and details. While the chatbots demonstrated a high degree of reliability, their validity-especially under high-threshold criterion-remains limited. Improvements in accuracy and comprehensiveness are necessary for more effective use in providing information about dental implants.