Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learning.

Area of Science:

• Artificial Intelligence
• Machine Learning
• Computer Vision

Background:

• Spiking neural networks (SNNs) excel at temporal sequence encoding for audio-visual feature extraction.
• Integrating SNNs (binary) with transformers (float-point) for temporal-semantic analysis presents challenges.

Purpose of the Study:

• To introduce a novel Spiking Tucker Fusion Transformer (STFT) for audio-visual zero-shot learning (ZSL).
• To enhance the joint exploration of temporal and semantic information in audio-visual data.

Main Methods:

• Developed a Spiking Tucker Fusion Transformer (STFT) integrating SNNs and transformers.
• Introduced a time-step factor (TSF) for dynamic synthesis of inference information.
• Proposed global-local pooling (GLP) to guide membrane potentials and reduce spike noise.
• Implemented a temporal-semantic Tucker fusion module for multi-scale fusion and second-order interaction maintenance.

Main Results:

• Achieved state-of-the-art performance on three benchmark datasets (VGGSound, UCF101, ActivityNet).
• Demonstrated significant harmonic mean (HM) improvements: 15.4% on VGGSound, 3.9% on UCF101, and 14.9% on ActivityNet.

Conclusions:

• The STFT effectively fuses temporal and semantic information from SNNs and transformers.
• The proposed methods address challenges in integrating binary SNNs with float-point transformers for ZSL.
• The approach offers a robust solution for audio-visual zero-shot learning tasks.