Advancing Human Motion Recognition with SkeletonCLIP++: Weighted Video Feature Integration and Enhanced Contrastive...

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Traditional human action recognition relies on label-based methods.
• Existing models may not fully capture nuanced human movements or semantic relevance.
• There is a need for advanced techniques to improve action differentiation.

Purpose of the Study:

• To introduce SkeletonCLIP++, an extended framework for human action recognition.
• To enhance the model's ability to utilize semantic information beyond simple labels.
• To improve the nuanced representation and differentiation of human actions in videos.

Main Methods:

• Implemented 'Weighted Frame Integration' (WFI) for nuanced video feature computation.
• Introduced 'Contrastive Sample Identification' (CSI) for enhanced action discrimination.
• Integrated the pre-trained 'BERT Text Encoder Integration' (BTEI) to refine performance.

Main Results:

• Demonstrated positive performance improvements on HMDB-51, UCF-101, and NTU RGB+D 60 datasets.
• Showcased particular effectiveness on smaller datasets.
• Achieved more detailed differentiation between closely related human actions.

Conclusions:

• SkeletonCLIP++ offers a refined approach to human action recognition.
• The model ensures semantic integrity and detailed differentiation in video data analysis.
• This work advances the field by incorporating semantic understanding into action recognition models.