An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits

Area of Science:

• Neuroscience
• Artificial Intelligence
• Signal Processing

Background:

• Audio-visual speech separation (AVSS) benefits from visual input, but optimizing combined auditory and visual processing remains a challenge.
• The brain's cortico-thalamo-cortical circuit demonstrates cross-modal sensory integration.
• Existing AVSS methods have limitations in efficiently utilizing multi-modal information.

Purpose of the Study:

• To propose a novel cortico-thalamo-cortical neural network (CTCNet) for enhanced audio-visual speech separation.
• To mimic the brain's sensory processing pathways for improved multi-modal fusion.
• To develop a more parameter-efficient AVSS model.

Main Methods:

• Developed CTCNet, a neural network with separate auditory and visual subnetworks for hierarchical representation learning.
• Implemented a thalamic subnetwork for fusing auditory and visual information via top-down connections, inspired by brain anatomy.
• Iteratively refined fused information through repeated transmission between cortical and thalamic subnetworks.

Main Results:

• CTCNet demonstrated superior performance on three benchmark speech separation datasets compared to existing AVSS methods.
• The proposed model achieved these results with significantly fewer parameters than current approaches.
• Experimental validation supports the efficacy of brain-inspired architectural design in AVSS.

Conclusions:

• The cortico-thalamo-cortical neural network (CTCNet) offers a significant advancement in audio-visual speech separation.
• Mimicking mammalian brain connectomics provides a promising avenue for developing more efficient and effective deep neural networks.
• CTCNet's success highlights the potential of neuro-inspired AI for complex signal processing tasks.