ViTPose++: Vision Transformer for Generic Body Pose Estimation.

Area of Science:

• Computer Vision
• Machine Learning

Background:

• Body pose estimation is crucial for various applications.
• Existing methods often involve complex architectures.
• Vision transformers offer a promising alternative for feature extraction.

Purpose of the Study:

• To evaluate the effectiveness of plain vision transformers for body pose estimation.
• To introduce ViTPose, a simple yet powerful baseline model.
• To explore the scalability and flexibility of vision transformer-based pose estimation.

Main Methods:

• Utilized a plain, non-hierarchical vision transformer as the encoder.
• Employed a lightweight decoder for keypoint decoding (top-down or bottom-up).
• Developed ViTPose++ incorporating knowledge factorization for heterogeneous keypoint categories.

Main Results:

• ViTPose demonstrates simplicity, scalability (up to 1B parameters), and flexibility.
• Achieved a new Pareto front for throughput and performance in pose estimation.
• ViTPose-G set a new record on the MS COCO test set.
• ViTPose++ achieved state-of-the-art performance across multiple human and animal keypoint detection datasets.

Conclusions:

• Plain vision transformers are surprisingly effective for body pose estimation.
• ViTPose offers a scalable, flexible, and high-performing solution.
• ViTPose++ advances the state-of-the-art in multi-task pose estimation.