C2F-TCN: A Framework for Semi- and Fully-Supervised Temporal Action Segmentation.

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Temporal action segmentation involves assigning labels to video frames for sequential actions.
• Existing methods often require extensive labeled data for accurate performance.

Purpose of the Study:

• To propose a novel encoder-decoder architecture, C2F-TCN, for improved temporal action segmentation.
• To develop model-agnostic temporal feature augmentation and explore unsupervised and semi-supervised learning strategies.

Main Methods:

• Developed the C2F-TCN architecture with a coarse-to-fine decoder ensemble.
• Implemented a novel, computationally inexpensive temporal feature augmentation using stochastic max-pooling.
• Introduced an unsupervised learning method leveraging feature clustering and multi-resolution decoder features.
• Proposed the Iterative-Contrastive-Classify (ICC) semi-supervised learning scheme.

Main Results:

• C2F-TCN achieved accurate and well-calibrated supervised results on three benchmark datasets.
• Demonstrated the flexibility of C2F-TCN for both supervised and representation learning.
• The unsupervised approach effectively learned frame-wise representations.
• The ICC semi-supervised method with 40% labeled data performed comparably to fully supervised methods.

Conclusions:

• C2F-TCN offers a flexible and effective framework for temporal action segmentation across various learning paradigms.
• The proposed feature augmentation and learning strategies enhance accuracy and reduce reliance on fully labeled data.
• Semi-supervised learning with ICC shows significant promise for efficient action segmentation with limited labels.