Decoding olfactory response from neurophysiological signal with a multi modal deep learning framework | JoVE Visualize

Area of Science:

Neuroscience
Sensory Perception
Computational Neuroscience

Background:

The temporal dynamics of the human olfactory system are vital for sensory perception.
Understanding neural features of olfactory perception from electroencephalography (EEG) requires analyzing temporal dynamics and integrating physiological signals like breathing.
Existing methods for decoding olfactory responses from EEG have limitations in capturing complex temporal and spatial neural information.

Purpose of the Study:

To introduce a novel multimodal deep learning framework, the Token Alignment and Cross-Attention Fusion network (TACAF), for enhanced olfactory EEG decoding.
To investigate the neural mechanisms underlying olfactory perception and adaptation using EEG and breathing signals.
To improve the accuracy and robustness of decoding olfactory responses by effectively fusing multimodal data.

Main Methods:

Development of the Token Alignment and Cross-Attention Fusion network (TACAF), a multimodal deep learning framework.
Utilizing wavelet features for time window selection and spectral analysis for EEG data representation.
Employing spatial learning modules for spatial feature extraction and multi-head self-attention for temporal dynamics.
Implementing the Temporal Token Semantic Alignment (TTSA) module to synchronize EEG and breathing data for fusion.
Collecting EEG and breathing recordings from 20 subjects exposed to pleasant and unpleasant odors.

Main Results:

The TACAF framework significantly outperforms existing methods in decoding olfactory EEG responses.
Analysis revealed that prolonged odor exposure induces olfactory adaptation, leading to decreased recognition performance.
The study successfully decoded neural responses to different odor types (pleasant vs. unpleasant).

Conclusions:

The TACAF network provides a powerful tool for decoding olfactory perception from EEG, outperforming current approaches.
Olfactory adaptation is a significant factor affecting odor recognition performance, highlighting the dynamic nature of sensory processing.
The findings offer valuable insights into the neural mechanisms of olfactory perception and adaptation, visualized through spatial topology and saliency maps.