Hybrid artificial intelligence framework for patient emotion recognition in dentistry

Emotional communication plays an important role in dentist-patient interactions, yet domain-specific resources for automated emotion recognition in dental consultations remain limited.

This study aimed to develop an emotional dental consultation corpus derived from restorative dentistry consultations and to evaluate a hybrid artificial intelligence (AI) framework for automated patient emotion recognition.

Audio recordings from 12 restorative dentistry consultations were transcribed and segmented into patient utterances. A total of 2,160 utterances were independently annotated into six emotion categories (Neutral, Fear, Sad, Happy, Angry, and Disgust) by three trained annotators using a standardized annotation framework. Inter-annotator reliability was assessed using Cohen's kappa coefficient. The resulting corpus was used to train a Bidirectional Long Short-Term Memory (BiLSTM) classifier and transformer-based baseline models, including BERT and RoBERTa. A hybrid framework was developed by applying large language model (LLM)-based confidence re-ranking to low-confidence BiLSTM predictions. Three open-source LLMs (Llama 3.1, DeepSeek-R1, and Qwen 2.5) were evaluated.

The developed corpus demonstrated high annotation reliability (κ = 0.89). Neutral (23.15%), Fear (19.44%), and Disgust (18.98%) were the most prevalent emotion categories. Fear was both one of the most frequently occurring emotions and the most challenging category to classify. The baseline BiLSTM model achieved a macro-F1 score of 0.87, whereas BERT and RoBERTa achieved macro-F1 scores of 0.89 and 0.91, respectively. The hybrid frameworks achieved numerically higher performance, with the BiLSTM + Qwen 2.5 framework obtaining the highest macro-F1 score of 0.92. However, the observed performance improvements were not statistically significant (all p > 0.05).

This study presents a novel emotion-annotated dental consultation corpus and demonstrates the feasibility of a confidence-based hybrid AI framework for automated patient emotion recognition in dentistry. Although the proposed framework achieved the highest numerical performance, the findings should be regarded as preliminary evidence of the potential utility of selective LLM-based re-ranking for recognizing emotionally ambiguous patient utterances.