Image-free single-pixel sensing for human pose estimation and parameter-efficient fine-tuning.

Area of Science:

• Computer Vision
• Machine Learning
• Signal Processing

Background:

• Current human pose estimation (HPE) methods require high-resolution images/videos, leading to high hardware costs and privacy concerns.
• Existing single-pixel sensing techniques for HPE lack sufficient accuracy and generalization capabilities.

Purpose of the Study:

• To develop a novel, image-free single-pixel sensing technique for efficient and robust 2D and 3D human pose estimation.
• To reduce computational and hardware costs associated with HPE while enhancing privacy protection.

Main Methods:

• Encoding scenes into a minimal number of measurements.
• Recovering intermediate feature representations from these measurements.
• Directly performing 2D and 3D HPE using a Vision Transformer-Huge (ViT-H) backbone with a parallel Adapter.

Main Results:

• Achieved 67.9 AP for 2D HPE on the COCO dataset at a 6.25% sampling rate, outperforming state-of-the-art CNN backbones.
• Demonstrated superior 3D HPE performance on the Human3.6M dataset compared to the SimpleBaseline3D model.
• Fine-tuned ViT-H with a parallel Adapter required training only 45.1% of parameters at 9.375% sampling rate, outperforming full-parameter training.

Conclusions:

• The proposed image-free single-pixel sensing technique offers a promising solution for accurate and privacy-preserving HPE.
• This approach significantly reduces hardware and computational costs, enabling efficient multi-task generalization.
• The novel fine-tuning strategy with a parallel Adapter in ViT-H enhances performance while minimizing trainable parameters.