Transformer-based fusion model for mild depression recognition with EEG and pupil area signals

Early detection and treatment are crucial for the prevention and treatment of depression; compared with major depression, current researches pay less attention to mild depression. Meanwhile, analysis of multimodal biosignals such as EEG, eye movement data, and magnetic resonance imaging provides reliable technical means for the quantitative analysis of depression. However, how to effectively capture relevant and complementary information between multimodal data so as to achieve efficient and accurate depression recognition remains a challenge. This paper proposes a novel Transformer-based fusion model using EEG and pupil area signals for mild depression recognition. We first introduce CSP into the Transformer to construct single-modal models of EEG and pupil data and then utilize attention bottleneck to construct a mid-fusion model to facilitate information exchange between the two modalities; this strategy enables the model to learn the most relevant and complementary information for each modality and only share the necessary information, which improves the model accuracy while reducing the computational cost. Experimental results show that the accuracy of the EEG and pupil area signals of single-modal models we constructed is 89.75% and 84.17%, the precision is 92.04% and 95.21%, the recall is 89.5% and 71%, the specificity is 90% and 97.33%, the F1 score is 89.41% and 78.44%, respectively, and the accuracy of mid-fusion model can reach 93.25%. Our study demonstrates that the Transformer model can learn the long-term time-dependent relationship between EEG and pupil area signals, providing an idea for designing a reliable multimodal fusion model for mild depression recognition based on EEG and pupil area signals.