Spatio-temporal bidirectional Long Short-Term Memory-based category decoding of natural dynamic facial expression...

Visual perceptual decoding of facial expressions is a key focus in affective neuroscience. Developing a mapping model between visual content and signals is crucial for decoding. Most previous visual decoding models focused on brain responses to static images, neglecting temporal-dynamic feature modeling. Additionally, they input all visual cortices as a whole into the model, overlooking that visual information flows bidirectionally between the lower and higher visual cortices based on bottom-up and top-down visual mechanisms, thus hard to capture bidirectional information between visual regions located in different spatial positions. Here, we present a spatio-temporal bidirectional long short-term memory-based model to decode 3 categories of facial expressions from multi-time functional magnetic resonance imaging data. Specifically, we used the spatio-temporal bidirectional long short-term memory module with the ability to simulate time series to grasp the temporal-dependence from visual cortices, and its forward and backward directions simulate bidirectional information flow between visual cortices to capture the bidirectional spatial information. Experimental outcomes indicate that the mean decoding accuracy employing beta estimates of multi-time response signals (Repetition Times（TR）1-6 1 to 6) from 5 participants is significantly higher than that of other time points signals, unidirectional connections, and publicly available models. These results reveal that our model captures temporal-dependencies and bidirectional spatial information from the visual cortices, enhancing decoding performance.