Artificial intelligence in maxillofacial trauma: expert ally or unreliable assistant?

Area of Science:

• Artificial Intelligence in Medicine
• Oral and Maxillofacial Surgery
• Medical Education Technology

Background:

• Large language models (LLMs) show promise in synthesizing complex medical information.
• Concerns exist regarding the accuracy and reliability of LLMs in specialized fields.
• Oral and maxillofacial traumatology lacks comprehensive evaluation of LLM capabilities.

Purpose of the Study:

• To evaluate the accuracy and reliability of ChatGPT-4o in oral and maxillofacial traumatology.
• To identify the limitations of current LLMs in this specialized domain.

Main Methods:

• 188 oral and maxillofacial trauma questions were curated.
• 30 questions were randomly selected and posed to ChatGPT-4o.
• Accuracy was rated on a 3-point Likert scale.
• Reliability was assessed using weighted kappa and ICC; internal consistency via Cronbach's alpha and McDonald's omega.

Main Results:

• Comprehensive and adequate response accuracy rates were 38% and 58%, respectively.
• Moderate reliability was indicated by weighted kappa (0.469) and ICC (0.503).
• Excellent and good internal consistency was observed (alpha=0.904, omega=0.860).

Conclusions:

• ChatGPT-4o shows potential as an adjunct tool for education and decision support in oral and maxillofacial traumatology.
• Current limitations necessitate further research and development.
• Future LLM enhancements and prompt engineering may improve clinical applicability and adherence to evidence-based standards.