Related Experiment Video
Updated: Mar 29, 2026
Exploring the Use of Isolated Expressions and Film Clips to Evaluate Emotion Recognition by People with Traumatic Brain Injury
Published on: May 15, 2016
eXCube2: Explainable Brain-Inspired Spiking Neural Network Framework for Emotion Recognition from Audio, Visual and
N K Kasabov1,2,3, A Yang4, Z Wang1
1Knowledge Engineering and Discovery Research Institute (KEDRI), School of Engineering, Computer and Mathematical Sciences, Auckland University of Technology (AUT), WZ Building, St. Paul Street, Auckland 1010, New Zealand.
This study introduces brain-inspired AI (BIAI) using spiking neural networks (SNNs) for emotional recognition. The eXCube2 framework achieves high accuracy in recognizing emotions from audio-visual data, offering improved explainability and adaptability.
Area of Science:
- Artificial Intelligence
- Computational Neuroscience
- Biomimetics
Background:
- Current AI for emotional state recognition often lacks explainability and adaptability.
- Spiking neural networks (SNNs) offer a biologically plausible approach for AI, mimicking brain function.
- Multimodal data integration (audio-visual) can enhance the accuracy of emotional state recognition.
Purpose of the Study:
- To introduce a biomimetic framework (eXCube2) with novel brain-inspired AI (BIAI) models for emotional state recognition.
- To develop SNN-based BIAI models using a human brain template for enhanced data processing.
- To evaluate the explainability, adaptability, and accuracy of the proposed framework on audio, visual, and multimodal data.
Main Methods:
- Developed the eXCube2 framework utilizing a 3D SNN architecture (NeuCube) structured on a human brain template.
- Trained BIAI models on spatio- and spectro-temporal data using brain-inspired learning rules.
- Tested models on benchmark audio-visual datasets for emotional state recognition, incorporating a "don't know" output.
Main Results:
- Achieved classification accuracy above 80% for single-modality data and 88.9% for multimodal audio-visual data.
- Demonstrated improved explainability and adaptability compared to existing AI systems.
- The brain template enabled effective tonotopic, stereo, and topographic mapping of audio-visual inputs.
Conclusions:
- The eXCube2 framework offers a novel, brain-inspired approach to emotional state recognition with high accuracy and enhanced AI characteristics.
- The SNN-based BIAI models show potential for applications in diverse fields, including healthcare, human-robot interaction, and gaming.
- This research advances the development of brain-inspired AI systems, paving the way for more efficient and explainable AI through neuromorphic computing.
