Audiovisual sequence memory enabled by a neurodynamic model with primary sensory connections

To enhance the reliability and stability of information acquisition, biological perception systems have gradually evolved from unimodal processing to multimodal integration mechanisms. Audio-visual cross-modal integration, in particular, has demonstrated robustness and generalization capabilities in key cognitive tasks such as speech recognition, emotion processing, and spatial localization. Moreover, increasing physiological evidence points to the existence of direct connections between primary perceptual layers during audiovisual information processing. However, how these connections support the integration, retrieval, and conflict detection in audiovisual sequence memory remains a central yet unresolved question in cognitive neuroscience. To address this issue, we draw on neural dynamics principles and incorporate the brain's memory connection mechanisms to develop a sequence memory model for audio-visual integration. At both the structural organization and computational process levels, the model enables self-organizing integration of multimodal information, supports bidirectional retrieval of semantic and content information, and possesses the capability to automatically detect inconsistencies between audio and visual inputs. Furthermore, the model reveals that, during multimodal sequence memory, the connection strength between primary sensory layers plays a critical role in cognitive performance and memory stability. In summary, this framework not only provides a computable theoretical model for understanding the neural mechanisms underlying audio-visual integration and its interaction with memory, but also offers new perspectives for developing multimodal intelligent systems with greater biological plausibility and interpretability. It thus addresses several key limitations of traditional deep learning approaches in both performance and mechanistic transparency.

Inspired by neurophysiology, we construct a sequence memory neurodynamic model that integrates visual and auditory processing, and validate its reliability through both micro- and macro-level experiments. We also explore novel functionalities of the model, such as bidirectional retrieval and automatic detection of audio-visual mismatches. Furthermore, the network is analyzed in terms of robustness and other performance metrics. This framework enables the development of next-generation multimodal agents with biological plausibility.

The online version contains supplementary material available at https://doi.org/10.1007/s11571-026-10545-w.