Local Dimension Enhancement Representation Learning for Skeleton-Based Action Segmentation

Area of Science:

• Computer Vision
• Machine Learning
• Artificial Intelligence

Background:

• Existing self-supervised learning methods for skeleton-based temporal action segmentation (TAS) often produce coarse or motion-insensitive representations.
• This leads to local dimension collapse, hindering accurate frame-level action prediction.

Purpose of the Study:

• To address local dimension collapse in self-supervised learning for TAS.
• To introduce a novel framework, Local Dimension Enhancement (LoDE), for improved skeleton-based action recognition.

Main Methods:

• Proposed the Local Dimension Enhancement (LoDE) framework utilizing local effective rank (LER) to measure and reduce dimension collapse.
• Introduced 'motion units' (temporal clips of skeleton frames) for fine-grained skeleton data modeling.
• Designed a multi-scale semantics module integrating frame-, sequence-, and motion unit-scale learning with LER-based regularization.

Main Results:

• LoDE effectively alleviates local dimension collapse by enriching local representation diversity.
• Demonstrated significant improvements in TAS performance compared to state-of-the-art methods.
• Achieved superior results on three large-scale untrimmed datasets: PKUMMD, TSU, and BABEL.

Conclusions:

• Motion unit-scale learning is crucial for alleviating local dimension collapse in TAS.
• The LoDE framework offers a promising direction for enhancing self-supervised learning in skeleton-based action recognition.
• The proposed methods lead to substantial gains in accuracy for dense frame-level prediction tasks.