Related Experiment Video
Updated: Jun 10, 2026
Virtual Agent for Real-Time Motivational Interviewing by Integrating Adaptive Nonverbal Behavior and Language Models
Published on: December 23, 2025
Role-based evaluation of artificial intelligence chatbot responses in orthodontic emergency scenarios: accuracy,
Mustafa Ozdemir1, Aysegul Gulec2
1Dentistry Faculty, Department of Orthodontics, Gaziantep University, Sehitkamil, Gaziantep, 27060, Turkey. dtmustafaozdemir@gmail.com.
Objectives:
This study investigated role-based differences in the accuracy, readability, understandability, and internal consistency of chatbot responses to orthodontic emergencies and examined the clinical implications of patient-oriented communication.
Methods:
Twenty-three standardized orthodontic emergency scenarios were presented to four chatbots- ChatGPT-4o, Claude 3 Opus, Microsoft Copilot, and Gemini 2.5-using patient and orthodontist roles. Response accuracy was evaluated by expert orthodontists and research assistants using a 3-point Likert scale, while readability, understandability, and internal consistency were assessed with the Atesman index, Sonmez formula, and Cronbach's α. Patient-role responses were descriptively analyzed using predefined communication dimensions to contextualize quantitative findings.
Results:
Significant chatbot-specific role-based differences were observed, with Claude 3 Opus (p = 0.001) and Gemini 2.5 (p = 0.023) showing higher accuracy in the orthodontist role. In the patient role, ChatGPT-4o and Claude 3 Opus showed the highest rates of correct information, while Claude 3 Opus had the highest rate in the orthodontist role. Patient-role responses were significantly more understandable than orthodontist-role responses (p < 0.05). ChatGPT-4o (α = 0.862) and Gemini 2.5 (α = 0.815) showed high internal consistency. Qualitative analysis indicated that patient-oriented responses frequently adopted a reassuring tone and emphasized temporary self-care strategies, potentially influencing perceived urgency in orthodontic emergencies.
Conclusions:
Chatbot performance varied according to user role. Patient-oriented responses were more understandable despite similar readability across models and could influence perceptions of urgency and professional responsibility, highlighting the need for cautious framing of chatbot-generated information for orthodontic emergency guidance.
Clinical Relevance:
Chatbots can provide preliminary information in orthodontic emergencies; however, due to limitations in accuracy and consistency, they should be used only as supportive tools and should not replace professional clinical judgment.
Related Concept Videos
01:26Types of Reports III: Telephone and Verbal Reports
Here's an overview of each type:
Telephone Orders
01:14SBAR II: Application of SBAR
SBAR Report from a Nurse to a Health Care Provider
S: "Hello, Dr. Smith. This is Jane, RN, from the Med Surg unit. I am calling to tell you about Ms. White in Room 210, who is experiencing increased pain and redness at her incision site. Her recent...
01:21Barriers to Effective Communication II
Cultural barriers:
Differences in values, beliefs, religion, knowledge, and tradition can significantly impact communication. Awareness of nonverbal cues is critical, especially when conversing with a patient from a different culture. What appears appropriate in one culture may be inappropriate in another.
Semantic barriers:
As a result of their tendency to use...