Related Experiment Video
Updated: Sep 16, 2025
Assessment and Communication for People with Disorders of Consciousness
Published on: August 1, 2017
A review of hybrid EEG-based multimodal human-computer interfaces using deep learning: applications, advances, and
Hyung-Tak Lee1,2, Miseon Shim3, Xianghong Liu4
1Department of Electronics and Information Engineering, Korea University, 2511, Sejong-ro, Jochiwon-eup, Sejong-si, 30019 Republic of Korea.
This review explores deep learning in multimodal human-computer interaction (HCI) using electroencephalography (EEG) and other biosignals. It highlights improved applications but notes challenges in real-time systems and explainable AI.
Area of Science:
- Neuroscience and Computer Science
- Focuses on the intersection of brain-computer interfaces and artificial intelligence.
Background:
- Human-computer interaction (HCI) increasingly uses multimodal systems for enhanced performance.
- Electroencephalography (EEG)-based systems combined with deep learning show promise for intuitive human-computer interactions.
- A systematic review is needed to consolidate findings on hybrid EEG-based multimodal HCI systems.
Purpose of the Study:
- To systematically review deep learning applications in hybrid EEG-based multimodal HCI systems.
- To analyze biosignal combinations, neural network architectures, fusion strategies, performance, and applications.
- To identify current challenges and future directions in the field.
Main Methods:
- Systematic review of 124 studies from 2016-2024.
- Database search using keywords: 'Deep Learning' AND 'EEG' AND ('fNIRS' OR 'NIRS' OR 'MEG' OR 'fMRI' OR 'EOG' OR 'EMG' OR 'ECG' OR 'PPG' OR 'GSR').
- Analysis of biosignal types, neural network architectures, fusion strategies, system performance, and applications.
Main Results:
- Electrooculography (EOG), electromyography (EMG), and functional near-infrared spectroscopy (fNIRS) are common complementary signals for EEG.
- Convolutional neural networks (CNNs) are widely used for feature extraction; early and intermediate fusion strategies are prevalent.
- Significant performance improvements observed in applications like sleep stage classification, emotion recognition, and mental state decoding.
Conclusions:
- EEG-based multimodal HCI systems demonstrate considerable potential for improving human-computer interaction.
- Key challenges include developing real-time online systems, improving signal synchronization, and increasing data availability.
- Advancements in explainable AI (XAI), portable systems, and data augmentation are crucial for future development.
More Related Videos
11:25Simultaneous Scalp Electroencephalography EEG, Electromyography EMG, and Whole-body Segmental Inertial Recording for Multi-modal Neural Decoding
Published on: July 26, 2013
10:51An Experimental Platform to Study the Closed-loop Performance of Brain-machine Interfaces
Published on: March 10, 2011