MMPI Net: A Novel Multimodal Model Considering the Similarities Between Perception and Imagination for Image Evoked...

Area of Science:

• Neuroscience
• Cognitive Science
• Machine Learning

Background:

• Non-invasive electroencephalography (EEG) is crucial for decoding cognitive functions like visual perception and imagination.
• Visual perception and imagination share neural circuits, but research often studies them in isolation.
• Multimodal approaches can leverage commonalities between cognitive processes for better insights.

Purpose of the Study:

• To propose a novel multimodal model, MMPI Net, for jointly decoding EEG signals of visual image perception and imagination.
• To address the limitations of single-process focused research by exploring shared neural patterns.
• To enhance the accuracy and effectiveness of cognitive decoding through a unified approach.

Main Methods:

• MMPI Net integrates four components: Primitive Feature Extraction (PFE), Cross-Semantic Feature Fusion (CSFF), Joint Semantic Feature Decoder (JSFD), and Semantic Classification (SC).
• An Improved Channel Attention Mechanism with parallel convolutional branches and Diverse Branch Block enhances PFE.
• Cross-attention fusion in CSFF captures intermodal information, while a Kolmogorov-Arnold Network in JSFD improves classification.

Main Results:

• MMPI Net demonstrated significant performance improvements on two public datasets.
• Average accuracy gains of 14.22% and 106.1% were achieved compared to single-process models.
• The model effectively decodes joint visual perception and imagination signals.

Conclusions:

• MMPI Net offers a powerful multimodal framework for decoding complex cognitive functions.
• Joint analysis of related cognitive processes like visual perception and imagination enhances decoding accuracy.
• This approach highlights the potential of multimodal EEG analysis in cognitive neuroscience.