Local pattern aware 3D video swin transformer with masked autoencoding for realtime augmented reality gesture...

Area of Science:

• Computer Vision
• Human-Computer Interaction
• Machine Learning

Background:

• Traditional Transformer models face challenges in spatio-temporal feature extraction and real-time performance for gesture interaction.
• Efficient and accurate gesture recognition is crucial for advancing augmented reality applications.

Purpose of the Study:

• To propose a real-time augmented reality gesture interaction algorithm leveraging Swin Transformer and masked autoencoder.
• To enhance spatio-temporal feature extraction, real-time performance, and overall accuracy in gesture recognition.

Main Methods:

• Utilized synthetic data generation for efficient 3D gesture annotation.
• Developed an image denoising model using maximum a posteriori probability with weighted Euclidean distance and structural similarity optimization.
• Integrated EfficientNet and Transformer models with skip connections and triplet attention for gesture detection and segmentation.
• Introduced local texture feature prior (RTHLBP) for improved accuracy.
• Employed a Vision Transformer (ViT) architecture with a masked autoencoder, dynamic weight fusion, and relative total variation map for gesture classification.
• Implemented a multi-core parallel computing strategy for real-time performance optimization.

Main Results:

• The proposed model achieved superior accuracy, F1 score, and mean intersection over union (MIoU) on the 4 GTEA sub-dataset compared to CNN, Transformer, MobileNet, and DenseNet models.
• Demonstrated significant improvements in performance, especially on smaller datasets.
• Showcased substantial reductions in computation time and maintained high computational efficiency with increased DSP cores.

Conclusions:

• The developed algorithm effectively addresses the limitations of traditional models in real-time augmented reality gesture interaction.
• The combination of advanced deep learning techniques and optimization strategies leads to state-of-the-art performance in gesture recognition and segmentation.
• The optimized real-time performance makes the algorithm suitable for practical augmented reality applications.