Evaluating haptic experience using EEG and deep learning across multiple modalities: linking stimulus and...

Area of Science:

• Neuroscience
• Human-Computer Interaction
• Machine Learning

Background:

• Traditional haptic interface evaluations use subjective self-reports, limiting objectivity and real-time adaptation.
• Cognitive haptic interfaces utilize neurophysiological measures like electroencephalography (EEG) and deep learning for objective assessment.
• A critical challenge lies in labeling neural responses: using physical stimulation (PS) parameters or self-reported (SR) perceptions.

Purpose of the Study:

• To systematically investigate the impact of PS- versus SR-based labeling on deep learning model performance for EEG-based haptic feedback.
• To compare the effectiveness of PS and SR labeling schemes across four haptic feedback modalities.
• To determine how labeling impacts model stability and accuracy in decoding neural responses to haptic stimuli.

Main Methods:

• Trained deep learning models (ATCNet, EEG Inception, EEG Conformer) using EEG data labeled by both PS parameters and SR perceptions.
• Evaluated model performance across four haptic modalities: delayed force-feedback (DFF), fingertip vibration feedback (FVF), upper-body vibration feedback (UVF), and fingertip thermal feedback (FTF).
• Employed a group-level leave-one-subject-out (LOSO) cross-validation strategy to assess model generalizability.

Main Results:

• Models trained with PS labeling consistently demonstrated more stable and higher performance compared to SR-labeled models across all tested modalities.
• The performance gains for PS-labeled models were most pronounced at near-perceptual-threshold stimulation levels, where SR labels exhibit greater inter-individual variability.
• EEG-based models aligned more closely with physical stimulation parameters than subjective reports, particularly in ambiguous perceptual conditions.

Conclusions:

• Physical stimulation (PS) labeling provides a more robust and objective approach for training deep learning models in cognitive haptic interfaces compared to self-reported (SR) labeling.
• PS-trained decoders serve as effective foundational representations for haptic feedback systems, adaptable with user-specific SR data.
• This research supports the use of PS-grounded models for developing more reliable and adaptive haptic interfaces, especially in challenging perceptual regimes.