Transformers with Joint Tokens and Local-Global Attention for Efficient Human Pose Estimation

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) have advanced 2D body pose estimation.
• Existing methods face challenges in balancing accuracy, computational efficiency, and robustness.
• CNNs are efficient but lack long-range dependency handling; ViTs capture dependencies but have high computational costs.

Purpose of the Study:

• To propose two novel Vision Transformer (ViT)-based models for accurate, efficient, and robust 2D body pose estimation.
• To address the trade-off between accuracy and efficiency in pose estimation.
• To enhance robustness across diverse scenarios including pose variations, lighting, and occlusions.

Main Methods:

• EViTPose: Utilizes learnable joint tokens to process a subset of body patches, enabling controllable accuracy-efficiency trade-offs.
• UniTransPose: Combines an efficient multi-scale transformer encoder with local/global attention and an efficient sub-pixel CNN decoder for multi-scale handling.
• Unified skeletal representation trained across multiple benchmarks for improved robustness.

Main Results:

• EViTPose achieved 30% to 44% reduction in computational complexity (GFLOPs) with minimal accuracy loss (0% to 3.5%).
• UniTransPose demonstrated accuracy improvements from 0.9% to 43.8% across six benchmarks.
• Both models significantly outperformed state-of-the-art methods in accuracy and efficiency.

Conclusions:

• The proposed ViT-based models offer a superior balance of accuracy, efficiency, and robustness in 2D body pose estimation.
• EViTPose provides explicit control over the accuracy-efficiency trade-off.
• UniTransPose achieves high accuracy and efficiency through its multi-scale transformer and CNN decoder design.