Multiscale Skeleton-Based Temporal Action Segmentation Using Hierarchical Temporal Modeling and Prediction Ensemble.

Area of Science:

• Computer Vision
• Machine Learning
• Human Action Recognition

Background:

• Skeleton-based temporal action segmentation (TAS) aims to decompose untrimmed skeleton sequences into distinct action segments.
• Existing TAS methods struggle with temporal scale variations, leading to over- or under-segmentation and high computational costs.
• Current approaches often employ complex parallel multiscale feature extractors and refinement modules.

Purpose of the Study:

• To propose a computationally efficient and accurate multiscale skeleton-based TAS (MSTAS) method.
• To address the challenges of temporal scale variance and over-segmentation/under-segmentation in TAS.
• To improve the robustness of action segmentation for ambiguous action instances.

Main Methods:

• Introduced Multiscale Skeleton-based TAS (MSTAS) comprising Temporal Probability Pyramid (TPP) and Smoothed Multiscale Ensemble (SME).
• TPP utilizes a U-shape hierarchical temporal pyramid to represent actions as multiscale probability distributions.
• SME averages probability distributions, incorporating label smoothing for dynamic confidence calibration across scales.

Main Results:

• Achieved state-of-the-art performance on four public datasets, including +1.1% accuracy and +2.8% F1@0.5 on the LARa dataset.
• Demonstrated significant reduction in computational overhead: 70% fewer parameters and 80% fewer GFLOPS.
• Showcased improved handling of ambiguous action instances through confidence calibration and efficient utilization of temporal scales.

Conclusions:

• MSTAS offers a superior balance between accuracy and computational efficiency for skeleton-based TAS.
• The proposed TPP and SME effectively mitigate multiscale challenges and improve action segmentation robustness.
• The U-shape pyramid architecture shows compatibility with existing refinement modules for enhanced motion representation extraction.