Hypergraph-Based Audio-Visual Fusion for Obstructive Sleep Apnea Severity Estimation During Wakefulness

Obstructive sleep apnea (OSA) is associated with psychophysiological impairments, and recent studies have shown the feasibility of using speech and craniofacial images during wakefulness for severity estimation. However, the inherent limitations of unimodal data constrain the performance of current methods. To address this, we proposed a novel hypergraph-based multimodal fusion framework (HMFusion) that integrates psychophysiological information from audio-visual data. Specifically, we employ long short-term memory (LSTM)-based encoders to extract modality-specific temporal dynamics from pre-trained audio-visual embeddings and remotely photoplethysmography (rPPG)-derived heart rate sequences. A hypergraph neural network is then utilized to capture critical cross-modal interactions for OSA severity estimation. Evaluation on a dataset of 159 participants from a clinical sleep center demonstrates that the proposed model achieves area under the receiver operating characteristic curves (AUCs) of 88.26%, 86.07%, and 85.29%, with corresponding F1-scores of 92.91%, 85.50%, and 85.30% at Apnea-Hypopnea Index (AHI) thresholds of 5, 15, and 30 events/hour, respectively, outperforming state-of-the-art approaches. This study highlights the potential of psychophysiological data in enhancing OSA severity estimation during wakefulness, offering new avenues for clinical research in this field.