Time-frequency feature calculation of multi-stage audiovisual neural processing via electroencephalogram microstates

Audiovisual (AV) perception is a fundamental modality for environmental cognition and social communication, involving complex, non-linear multisensory processing of large-scale neuronal activity modulated by attention. However, precise characterization of the underlying AV processing dynamics remains elusive.

We designed an AV semantic discrimination task to acquire electroencephalogram (EEG) data under attended and unattended conditions. To temporally resolve the neural processing stages, we developed an EEG microstate-based analysis method. This involved segmenting the EEG into functional sub-stages by applying hierarchical clustering to global field power-peak topographic maps. The optimal number of microstate classes was determined using the Krzanowski-Lai criterion and Global Explained Variance evaluation. We analyzed filtered EEG data across frequency bands to quantify microstate attributes (e.g., duration, occurrence, coverage, transition probabilities), deriving comprehensive time-frequency features. These features were then used to classify processing states with multiple machine learning models.

Distinct, temporally continuous microstate sequences were identified characterizing attended versus unattended AV processing. The analysis of microstate attributes yielded time-frequency features that achieved high classification accuracy: 97.8% for distinguishing attended vs. unattended states and 98.6% for discriminating unimodal (auditory or visual) versus multimodal (AV) processing across the employed machine learning models.

Our EEG microstate-based method effectively characterizes the spatio-temporal dynamics of AV processing. Furthermore, it provides neurophysiologically interpretable explanations for the highly accurate classification outcomes, offering significant insights into the neural mechanisms underlying attended and unattended multisensory integration.