Facial Emotion Recognition via Fusion of Deep and Handcrafted Features

Facial emotion recognition plays an important role in affective computing and human-computer interaction. Although convolutional neural network (CNN)-based methods have demonstrated remarkable performance, deep features alone may not sufficiently capture subtle geometric deformations and local texture variations, particularly under limited training data and challenging real-world conditions. To address this limitation, this study proposes a hybrid framework that integrates CNN-based deep features with handcrafted geometric and texture features. Specifically, 17 landmark-based angular features extracted from the eyebrows, eyes, nose, and mouth are combined with histogram of oriented gradients (HOG) features extracted from the nose and mouth regions through feature-level concatenation. The proposed method was extensively evaluated on three controlled datasets (JAFFE, CK+, and KDEF) and two large-scale in-the-wild datasets (RAF-DB and AffectNet). Five-fold cross-validation, leave-one-subject-out cross-validation, statistical significance analysis using paired t-tests, computational efficiency analysis, and comparisons with conventional handcrafted methods, standard CNN models, transfer learning-based methods, and recent hybrid feature-fusion methods were performed to comprehensively validate the proposed approach. Experimental results demonstrated consistent improvements across different datasets, evaluation protocols, and CNN backbone networks while maintaining a favorable balance between recognition performance and computational efficiency. These findings demonstrate that handcrafted geometric and local texture features effectively complement CNN-based deep representations, providing a robust and generalizable framework for facial emotion recognition across both controlled and large-scale in-the-wild datasets.