Dual-stream interactive mechanism with multi-modal hierarchical aggregation transformer for gait recognition

Area of Science:

• Computer Science
• Biometrics
• Artificial Intelligence

Background:

• Current gait recognition methods are often unimodal, limiting performance.
• Multimodal gait recognition faces challenges in data integration, fusion, and spatio-temporal information utilization.
• Existing techniques struggle to capture long-range dependencies and fine-grained dynamic features in multimodal settings.

Purpose of the Study:

• To propose GaitSMAT, a novel network for multimodal gait recognition.
• To address limitations in data integration, feature fusion, and spatio-temporal information capture.
• To leverage complementary advantages of multimodal gait data for enhanced recognition.

Main Methods:

• GaitSMAT integrates silhouette and heatmap data using a Dual-Stream Interactive Mechanism (DSM) and Multi-modal Hierarchical Aggregation Transformer (MHAT).
• DSM enables spatial feature interaction, long-range dependency capture, and enhanced representational capacity via bidirectional exchange and adaptive scaling.
• MHAT facilitates dynamic cross-modal feature interactions, adaptively modulating feature importance and enhancing robustness.

Main Results:

• GaitSMAT achieves state-of-the-art (SOTA) performance on GREW, Gait3D, and SUSTech1K datasets.
• Demonstrates significant improvements over existing multimodal gait recognition approaches.
• Exhibits superior robustness and accuracy, particularly in complex environments.

Conclusions:

• GaitSMAT presents a novel technical framework for multimodal gait recognition.
• The proposed method effectively overcomes limitations of existing unimodal and multimodal approaches.
• Offers substantial implications for enhancing the performance and practicality of gait recognition systems.