Explainability guided feature selection for speech emotion recognition using SHAP and permutation importance

Speech Emotion Recognition (SER) systems commonly rely on high-dimensional handcrafted acoustic feature sets, which increase computational complexity, risk overfitting, and reduce interpretability. To address these challenges, we propose an explainability-driven, redundancy-aware feature selection framework that integrates SHAP-based attribution with permutation-based performance importance through a unified rank-sum criterion. A correlation-controlled redundancy penalty is further introduced to mitigate multicollinearity while preserving acoustically meaningful descriptors. The framework operates entirely in the original feature space, enabling transparent and interpretable model refinement. Experiments were conducted on four widely used emotional speech corpora (CREMA-D, RAVDESS, SAVEE, and TESS) under within-corpus evaluation settings. To assess the generalizability of the proposed framework, experiments were performed using multiple classifiers, including XGBoost, Support Vector Machine (SVM), and Random Forest (RF). Using the XGBoost classifier, the proposed framework achieved accuracies of 0.600 on CREMA-D, 0.752 on RAVDESS, 0.726 on SAVEE, and 0.998 on TESS, while reducing the feature space by up to 40-70%. Macro-F1 and UAR metrics exhibit similar trends, confirming stable performance gains. Additional experiments using SVM and RF further indicate that the proposed feature selection framework generalizes across different classifiers while maintaining competitive performance. The findings indicate that explainability can serve not only as a post-hoc interpretation tool but as an active optimization mechanism for compact and interpretable SER models.