Spatio-Temporal Transformer with Kolmogorov-Arnold Network for Skeleton-Based Hand Gesture Recognition

Area of Science:

• Computer Vision
• Machine Learning
• Human-Computer Interaction

Background:

• Manual feature engineering for gesture recognition is subjective and lacks robustness.
• Existing deep learning models often neglect crucial spatial-temporal and structural hand joint information.
• Capturing long-range dependencies between non-adjacent hand joints is vital for accurate recognition.

Purpose of the Study:

• To propose an advanced skeleton-based hand gesture recognition framework, ST-KT.
• To effectively model both spatial and temporal dependencies within human hand joint data.
• To leverage the strengths of graph convolutional networks and transformer architectures enhanced with Kolmogorov-Arnold Networks (KAN).

Main Methods:

• The ST-KT framework integrates spatio-temporal graph convolution network (ST-GCN) modules and a KAN-based transformer.
• ST-GCN modules (comprising spatial graph convolution network and temporal convolution network) extract initial skeleton sequence features.
• A spatio-temporal position embedding method enriches node representations with identity and temporal context, while KAN-Transformers capture intricate joint relationships.

Main Results:

• The proposed ST-KT method achieved high accuracy on challenging datasets: 97.5% on SHREC'17 and 94.3% on DHG-14/28.
• The framework effectively captures dynamic skeleton changes and complex inter-joint relationships.
• The integration of KAN within the transformer enhanced nonlinear modeling capabilities for richer feature extraction.

Conclusions:

• The ST-KT framework demonstrates superior performance in skeleton-based dynamic hand gesture recognition.
• The method successfully addresses limitations of previous approaches by incorporating spatio-temporal dynamics and long-range joint dependencies.
• This research offers a robust and accurate solution for advanced human-computer interaction applications.