Comparing the performance of conversational artificial intelligence models in answering patient questions about...

Area of Science:

• Artificial Intelligence in Healthcare
• Oral and Maxillofacial Surgery
• Patient Education

Background:

• Dental implants are a common solution for tooth loss.
• Patients frequently seek information about dental implants, including suitability, risks, and post-operative care.
• AI-powered chatbots are increasingly used for health information dissemination.

Purpose of the Study:

• To compare the reliability and usefulness of three AI chatbots (ChatGPT-4.5, DeepSeek, Claude Sonnet 3.7) in answering patient questions about dental implants.
• To evaluate chatbot responses against the consensus of oral and maxillofacial surgeons.

Main Methods:

• 27 patient-focused questions regarding dental implants were curated.
• AI chatbot responses were assessed for reliability and usefulness by two independent, blinded oral and maxillofacial surgeons.
• Responses were evaluated using a modified three-point Likert scale, with inter-rater reliability assessed via Cronbach's alpha.

Main Results:

• No statistically significant differences were found in the overall reliability or usefulness of answers provided by ChatGPT-4.5, DeepSeek, and Claude Sonnet 3.7 (p > 0.05).
• Claude 3.7 Sonnet provided longer, more detailed answers.
• ChatGPT-4.5 used fewer medical terms, and DeepSeek demonstrated higher Flesch Reading Ease scores compared to ChatGPT-4.5, though significant differences in avoiding medical jargon were noted (p < 0.001).

Conclusions:

• Widely used AI chatbots demonstrate comparable reliability and usefulness in addressing common patient queries about dental implants.
• These AI tools show potential as supplementary resources for patient education regarding dental implant procedures.
• Further research into AI's role in patient communication within oral surgery is warranted.