ECG-EmotionNet: Nested Mixture of Expert (NMoE) Adaptation of ECG-Foundation Model for Driver Emotion Recognition

Driver emotion recognition plays a crucial role in driver monitoring systems, enhancing human-autonomy interactions and the trustworthiness of Autonomous Driving (AD). Various physiological and behavioural modalities have been explored for this purpose, with Electrocardiogram (ECG) emerging as a standout choice for real-time emotion monitoring, particularly in dynamic and unpredictable driving conditions. Existing methods, however, often rely on multichannel ECG signals recorded under static conditions, limiting their applicability in real-world dynamic driving scenarios. To address this limitation, the paper introduces ECG-EmotionNet, a novel architecture designed specifically for emotion recognition in dynamic driving environments. ECG-EmotionNet is constructed by adapting a recently introduced ECG Foundation Model (FM) and uniquely employs single-channel ECG signals, ensuring computational efficiency. Unlike conventional adaptation methods such as full fine-tuning, linear probing, or low-rank adaptation, we propose an intuitively pleasing alternative, referred to as the Nested Mixture of Experts (NMoE) adaptation. More precisely, each transformer layer of the underlying foundation model (FM), along with the Short-Time Fourier Transform (STFT) of the raw ECG signal, is treated as a separate expert. The embeddings extracted from these experts are fused using trainable weights within a gating mechanism. This approach enhances the representation of both global and local ECG features, while maintaining computational efficiency. The effectiveness of the proposed ECG-EmotionNet architecture is evaluated using a recently introduced and challenging driver emotion monitoring dataset. The proposed architecture outperforms its counterparts, achieving an average subject-level classification accuracy of 70.00% and F1 score of 69.26%, across four emotional states: anger, fear, sadness, and surprise.