Unleashing the Power of Text-to-Image Diffusion Models for Category-Agnostic Pose Estimation

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Category-Agnostic Pose Estimation (CAPE) faces generalization challenges due to limited labeled data in few-shot settings.
• Existing methods often require extensive base-category annotated data, limiting their applicability to unseen object categories.

Purpose of the Study:

• To develop a novel framework, Prompt Pose Matching (PPM), for effective CAPE in a base-category-free setting.
• To leverage off-the-shelf text-to-image diffusion models for improved keypoint detection in unseen object categories.

Main Methods:

• PPM learns pseudo prompts from few-shot examples using text-to-image diffusion models to capture keypoint semantics.
• A category-agnostic pre-training strategy and Foreground-Aware Region Aggregation (FARA) module provide robust initialization and supervision.
• A Foreground-Guided Attention Refinement (FGAR) module enhances cross-attention for accurate keypoint localization, and Prompt Ensemble Inference (PEI) enables efficient joint prediction.

Main Results:

• The proposed PPM framework demonstrates strong performance in CAPE without relying on base-category annotated data.
• Learned pseudo prompts effectively capture semantic information for keypoint localization in unseen categories.
• The integration of FARA and FGAR modules ensures reliable prompt pre-training and accurate keypoint detection.

Conclusions:

• PPM offers a powerful and flexible approach to category-agnostic pose estimation, particularly in data-scarce scenarios.
• The framework successfully overcomes the limitations of traditional methods by operating in a base-category-free manner.
• This work highlights the potential of text-to-image diffusion models in advancing pose estimation research for novel object categories.