Relation-Based Associative Joint Location for Human Pose Estimation in Videos.

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Deep learning methods for video-based human pose estimation (VHPE) often implicitly model joint interactions.
• Existing approaches rely on expanded receptive fields or manual graph designs, limiting flexibility.

Purpose of the Study:

• To propose a novel, lightweight, and plug-and-play Joint Relation Extractor (JRE) for explicitly modeling joint relationships in VHPE.
• To develop a Relation-based Pose Semantics Transfer Network (RPSTN) that leverages JRE and temporal continuity for improved pose estimation.

Main Methods:

• The JRE takes pseudo heatmaps as input to automatically learn correlations between any two joints.
• A Joint Relation Guided Pose Semantics Propagator (JRPSP) transfers pose semantics across frames, aiding occluded joint inference.
• The RPSTN integrates JRE with temporal semantic continuity modeling.

Main Results:

• The proposed RPSTN achieves state-of-the-art or competitive performance on multiple VHPE datasets (Penn Action, Sub-JHMDB, PoseTrack2018, HiEve).
• The JRE module enhances backbone performance on image-based pose estimation tasks (COCO2017).
• The method effectively infers invisible or occluded joints by learning joint correlations.

Conclusions:

• The JRE offers an effective way to explicitly model joint relationships, advancing VHPE.
• The RPSTN framework demonstrates superior performance by combining explicit relation modeling with temporal dynamics.
• The approach shows promise for both video-based and image-based human pose estimation tasks.