Multimodal Contrastive Learning for Cybersickness Recognition Using Brain Connectivity Graph Representation.

Area of Science:

• Neuroscience
• Computer Science
• Virtual Reality

Background:

• Cybersickness negatively impacts virtual reality (VR) user experience and immersion.
• Current cybersickness detection methods struggle with accuracy due to limited inter-modal relationship modeling.

Purpose of the Study:

• To develop an advanced multimodal contrastive learning method for improved cybersickness recognition.
• To enhance the modeling of relationships between physiological, visual, and motion data for better detection.

Main Methods:

• Introduced Brain Connectivity Graph Representation (BCGR) to capture cross-modal connectivity patterns.
• Developed E-BCGR (EEG), MV-BCGR (video/motion), and S-BCGR (standardized decomposition).
• Proposed a connectivity-constrained contrastive fusion module using graph contrastive learning.

Main Results:

• The proposed method significantly outperforms existing state-of-the-art approaches.
• Achieved superior performance across accuracy, sensitivity, specificity, and AUC metrics.
• Demonstrated the effectiveness of BCGR and the contrastive fusion module.

Conclusions:

• The novel multimodal contrastive learning approach effectively identifies cybersickness.
• The BCGR representation and connectivity-constrained fusion enhance detection accuracy.
• The developed dataset and method pave the way for future research in VR cybersickness mitigation.