Evaluation of Large Language Model-Based Chatbots for Dental Trauma Management: A Comparative Study Based on...

Area of Science:

• Dental Traumatology
• Artificial Intelligence in Healthcare
• Natural Language Processing

Background:

• Large language model (LLM)-based chatbots offer potential tools for healthcare information retrieval.
• Evaluating the performance of these AI tools in specialized medical fields like dentistry is crucial.
• Traumatic dental injuries (TDIs) require accurate and timely information for effective management.

Purpose of the Study:

• To compare the accuracy, consistency, readability, and information quality of three leading LLM-based chatbots (ChatGPT-5, Claude AI, Perplexity) for TDIs.
• To assess the suitability of these AI tools for clinical decision support in dental trauma.
• To provide insights into selecting appropriate AI tools based on specific use cases in dental trauma management.

Main Methods:

• Accuracy and consistency were evaluated using 40 true/false statements submitted thrice to each chatbot.
• Readability, understandability, actionability, and information reliability of responses to 25 open-ended case-based questions were assessed.
• Quantitative metrics including Flesch Reading Ease (FRE) and mDISCERN scores were utilized.

Main Results:

• Perplexity demonstrated the highest accuracy for true/false questions, followed by Claude AI and ChatGPT-5.
• ChatGPT-5 achieved the best readability scores for open-ended responses.
• Perplexity excelled in understandability and actionability, while Claude AI showed superior information reliability.

Conclusions:

• LLM-based chatbots show a complementary role in dental trauma management, with varying strengths.
• Perplexity, Claude AI, and ChatGPT-5 offer distinct advantages for different aspects of TDI information retrieval.
• Strategic selection of AI tools based on specific needs, coupled with essential human clinical oversight, is recommended.