Align and Fuse: A Transformer-Based Framework for EEG-Augmented Visual Recognition

Area of Science:

• Computational Neuroscience
• Computer Vision
• Machine Learning

Background:

• Integrating electroencephalography (EEG) with computational vision for robust visual recognition is promising.
• Challenges exist in supporting mixed-granularity recognition due to heterogeneous EEG and visual data feature spaces.

Purpose of the Study:

• To develop a unified framework for robust visual recognition integrating human neural signals and computational vision.
• To address the challenge of mixed-granularity recognition by aligning and fusing heterogeneous feature spaces.

Main Methods:

• Proposed a two-stage Transformer-based framework named "Align and Fuse."
• Stage 1: Constructed a shared semantic space using hardness-aware multimodal supervised contrastive loss with Hard Negative Weighting.
• Stage 2: Employed a multimodal Transformer with co-attention for feature fusion and classification.

Main Results:

• Achieved 91.12% Top-1 accuracy on the 80-class EEG-ImageNet benchmark, outperforming vision-only and standard Transformer baselines.
• Reached 95.82% accuracy on the 40-class EEGCVPR dataset and 90.92% average Top-1 accuracy on unseen subjects.
• Grad-CAM analysis indicated that aligned EEG signals direct model attention to relevant visual regions.

Conclusions:

• The proposed framework effectively integrates complementary information from EEG and visual data for enhanced recognition.
• The system functions as a human-in-the-loop EEG-augmented recognition system, requiring EEG features at inference time.