Reliability of Large Language Model-Based Chatbots Versus Clinicians as Sources of Information on Orthodontics: A...

Area of Science:

• Dental Science
• Artificial Intelligence

Background:

• Large Language Model (LLM) chatbots are increasingly used for information retrieval.
• Their reliability in specialized fields like orthodontics requires thorough evaluation.

Purpose of the Study:

• To assess the reliability of LLM chatbots as information sources in orthodontics.
• To compare chatbot responses with those of general dental practitioners (GDPs) and orthodontic specialists (Os).

Main Methods:

• Eight true/false orthodontic questions were posed to five leading chatbots.
• Chatbot response consistency was measured using Cronbach's alpha.
• Chatbot answers were compared to clinician responses via Chi-squared tests (p < 0.05).
• Educational value was rated using the Global Quality Scale (GQS).

Main Results:

• All chatbots demonstrated high response consistency (alpha > 0.80).
• Significant differences (p < 0.05) were observed between chatbot and clinician responses for most questions.
• DeepSeek achieved the highest GQS score (median 4.00), while Copilot had the lowest (median 2.00).

Conclusions:

• LLM chatbots provide consistent, yet often divergent, information compared to dental professionals.
• Chatbots can offer valuable orthodontic insights but may be unreliable for controversial subjects.