EffortNet: A Deep Learning Framework for Objective Assessment of Speech Enhancement Technologies Using EEG-Based...

This paper presents EffortNet, a novel deep learning framework for decoding listening effort at the individual level from electroencephalography (EEG) during speech comprehension. Quantifying listening effort remains a significant challenge in speech-hearing research. We collected 64-channel EEG data from 122 participants during speech comprehension under four conditions: clean, noisy, MMSE-enhanced, and Transformer-enhanced speech. Statistical analyses confirmed that alpha power (8-13 Hz) was significantly higher during noisy speech processing compared with clean or enhanced conditions, confirming its validity as an objective biomarker of listening effort. To address the substantial inter-individual variability in EEG signals, EffortNet integrates three complementary learning paradigms: self-supervised learning to leverage unlabeled data, incremental learning for progressive adaptation to individual characteristics, and transfer learning for efficient knowledge transfer to new subjects. Our experimental results demonstrate that EffortNet achieves ${80}.{9}$ % classification accuracy with only 40% of the training data from new subjects, significantly outperforming conventional CNN ( ${62}.{3}$ %) and STAnet ( ${61}.{1}$ %) models. The probability-based metric derived from our model revealed that Transformer-enhanced speech elicited neural responses more similar to clean speech than MMSE-enhanced speech. This neural pattern aligns with objective metrics, but contrasts with subjective intelligibility ratings. This dissociation underscores the value of objective neural markers in evaluating hearing technologies, such as headsets, hearables, and assistive listening devices.