Development and Evaluation of a GPT4-Based Orofacial Pain Clinical Decision Support System

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Diagnostic Decision Support

Background:

• Orofacial pain (OFP) presents complex diagnostic challenges, often resulting in misdiagnosis.
• Artificial intelligence (AI), specifically large language models (LLMs), offers potential as a diagnostic aid.

Purpose of the Study:

• To evaluate GPT4's diagnostic accuracy for OFP as a clinical decision support system (CDSS).
• To compare GPT4's performance against clinicians, experts, medical students, and general practitioners.

Main Methods:

• 100 anonymized OFP case descriptions were used to prompt GPT4 for diagnoses.
• Diagnoses were evaluated against gold-standard diagnoses using the International Classification of Orofacial Pain (ICOP) criteria.
• GPT4's performance was compared to human evaluators on a subset of 24 cases.

Main Results:

• GPT4 achieved 52% overall diagnostic accuracy at ICOP level 3 and provided accurate differential diagnoses in 80% of cases.
• GPT4's performance was comparable to non-experts but lower than clinical experts.
• High accuracy was observed in specific conditions like trigeminal neuralgia (81% at level 3).

Conclusions:

• GPT4 demonstrates potential as a CDSS for OFP, improving diagnostic accuracy and differential diagnoses.
• GPT4 can augment diagnostic workflows, especially in primary care and education.
• Clinical integration requires rigorous validation, guidelines, and professional oversight for patient safety.