A2Net: Affiliation Alignment Networks for Whole-Body Pose Estimation With Vision--Language Models

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Whole-body pose estimation predicts human keypoints but suffers from scale variation and semantic ambiguity.
• Existing multiscale feature extraction methods fail to resolve semantic ambiguity in small body parts.

Purpose of the Study:

• To propose the affiliation alignment network (A2Net) for enhanced whole-body pose estimation.
• To overcome scale variation and semantic ambiguity issues in keypoint localization.

Main Methods:

• Developed A2Net utilizing vision-language hierarchical affiliations.
• Constructed a multisemantic hierarchical language latent space via Text Affiliation Injection.
• Employed optimal transport (OT) to align image and text features across hierarchical levels, creating a scale-independent latent space.

Main Results:

• A2Net demonstrated improved performance in whole-body pose estimation.
• The model effectively addressed challenges posed by image scale variations and small-scale semantic ambiguity.
• Experimental results on two datasets showed competitive performance against state-of-the-art methods.

Conclusions:

• A2Net offers a novel approach to whole-body pose estimation by leveraging vision-language alignment.
• The proposed method successfully mitigates scale variation and semantic ambiguity, leading to more accurate keypoint localization.