Modulation-Based Feature Extraction for Robust Sleep Stage Classification Across Apnea-Based Cohorts

Automated sleep staging remains challenging due to the transitional nature of certain sleep stages, particularly N1. In this paper, we explore modulation spectrograms for automatic sleep staging to capture the transitional nature of sleep stages and compare them with conventional benchmark features, such as the Short-Time Fourier Transform (STFT) and the Continuous Wavelet Transform (CWT). We utilized a single-channel EEG (C4-M1) from the DREAMT dataset with subject-independent validation. We stratify participants by the Apnea-Hypopnea Index (AHI) into Normal, Mild, Moderate, and Severe groups to assess clinical generalizability. Our modulation-based framework significantly outperforms STFT and CWT in the Mild and Severe cohorts, while maintaining comparable high performance in the Normal and Moderate AHI groups. Notably, the proposed framework maintained robust performance in severe apnea cohorts, effectively mitigating the degradation observed in standard time-frequency baselines. These findings demonstrate the effectiveness of modulation spectrograms for sleep staging while emphasizing the importance of medical stratification for reliable outcomes in clinical populations.