An attention-based multimodal deep learning framework integrating EEG and ECG for enhanced stress detection

Chronic stress is an important threat in Public Health, as it negatively impacts both the Body and Mind. Current methods for measuring and identifying stress rely largely on individuals providing subjective assessments or measuring isolated physiological parameters, thereby limiting the accuracy and consistency of these approaches. This study proposes a novel approach to objectively measuring an individual’s level of stress, by combining deep transfer learning methods for detecting psychological stress measured using Electroencephalogram (EEG) and Electrocardiogram (ECG) data. More specifically, this new method uses three pre-trained neural network backbones—VGG16, EfficientNetB0, and ResNeXt50—utilized together, to create a unified system capable of merging information from multiple streams of data in real-time. EEG data is converted to time-frequency maps using wavelet transformations and ECG data uses time-series variability (i.e. patterns of how the heart beats) combined with raw, unfiltered data. An advanced fusion layer uses attention weights to intelligently combine these two data sources, allowing for improved accuracy of stress assessment. Using the WESAD and CASE datasets, both of which were collected from 35 subjects while they were in a neutral (control), tense, and positive state, our method performs at 95.7% accuracy in identifying between these three conditions, which is significantly greater than the accuracy rates of either the EEG-only (82.3%) or ECG-only (85.6%) methods or individual networks. Furthermore, this system is highly flexible and has demonstrated the capability to successfully operate across numerous testing conditions, while additionally demonstrating that EEG signals enhance ECG stress assessment and vice versa. Therefore, this new approach provides a highly reliable way to support medical diagnosis, employee wellness programs, and personalized psychological support.