Hybrid fNIRS-EEG based classification of auditory and visual perception processes

Area of Science:

• Neuroscience
• Human-Computer Interaction (HCI)
• Signal Processing

Background:

• Identifying user's current information processing modality is crucial for effective multimodal HCI.
• Reducing user workload in HCI systems can be achieved by selecting complementary output modalities.
• Existing methods may lack the precision to differentiate between simultaneous cognitive processes.

Purpose of the Study:

• To develop a hybrid Brain-Computer Interface (BCI) for discriminating and detecting visual and auditory stimulus processing.
• To evaluate the performance of the hybrid BCI system using Electroencephalography (EEG) and functional Near Infrared Spectroscopy (fNIRS).
• To assess the contribution of individual signal types and their fusion for improved classification accuracy.

Main Methods:

• Developed a hybrid BCI system integrating EEG and fNIRS.
• Collected data from 12 subjects processing visual and auditory stimuli.
• Performed cross-validation to evaluate classification accuracy for subject-dependent and subject-independent conditions.

Main Results:

• Subject-dependent systems achieved 97.8% accuracy in discriminating visual from auditory perception.
• Up to 94.8% accuracy was obtained for detecting modality-specific processes.
• Subject-independent classification reached up to 94.6% and 86.7% for the respective conditions, with classifier fusion significantly increasing accuracy.

Conclusions:

• The hybrid EEG-fNIRS BCI effectively discriminates between visual and auditory processing.
• The system demonstrates potential for real-time workload management in multimodal HCI.
• Fusion of EEG and fNIRS signals enhances classification performance, highlighting the benefits of multimodal sensing.