Dual-view fusion of log-mel spectrogram encodings for dysarthria detection: use in the context of awake brain surgery

Intraoperative speech monitoring during awake glioma surgery is critical to detect stimulation-induced motor speech impairments such as dysarthria. However, automatic detection from short speech segments remains challenging due to limited annotations, strong acoustic variability, and transient impairments in operating-room conditions. This study investigates whether combining complementary encodings of the same log-mel representation can improve robustness under such constraints.

We propose a dual-branch EfficientNet-B0 architecture that processes two complementary views of each log-mel spectrogram: (i) a numeric amplitude-preserving representation stored as NPY, and (ii) an RGB spectrogram image stored as PNG using a fixed colormap. The resulting embeddings are fused using four strategies: naïve concatenation, attention-based fusion, and gated fusion (sigmoid and GRU-style). Models are evaluated under two settings: (i) intra-corpus group-wise cross-validation on TORGO (English, controlled) and DATABRASE (French, intraoperative), and (ii) cross-corpus transfer (TORGO $\to$ DATABRASE) to quantify domain shift. In addition, we evaluate DATABRASE with and without Demucs-based denoising to analyze the impact of operating-room noise.

In intra-corpus evaluation, adaptive fusion strategies provide consistent benefits on TORGO, with gated fusion achieving the highest AUC (0.852±0.087). On raw DATABRASE, the PNG unimodal baseline yields the best mean performance (UAR 0.830±0.132; AUC 0.889±0.109), whereas denoising altered the model ranking and improved the performance of attention-based fusion (UAR 0.854±0.106). Cross-corpus testing reveals a marked degradation for all methods, confirming a severe domain shift between TORGO and intraoperative speech.

Dual-encoding fusion can enhance dysarthria detection when training and testing conditions are aligned and noise is controlled, but robust cross-corpus generalization remains unresolved. These results highlight both the potential and current limitations of spectrogram-based deep models for intraoperative speech monitoring, motivating future work on domain-adaptive and self-supervised learning tailored to clinical recordings.