End-to-End Streaming Video Temporal Action Segmentation With Reinforcement Learning

Area of Science:

• Computer Science
• Artificial Intelligence
• Machine Learning

Background:

• Temporal action segmentation (TAS) is crucial for video understanding but typically operates offline.
• Existing TAS methods struggle with online scenarios due to reliance on complete data and multimodal features.
• Streaming temporal action segmentation (STAS) extends TAS to online settings, classifying frames sequentially.

Purpose of the Study:

• To address the inadequate attention and poor performance of existing methods on the STAS task.
• To analyze the fundamental differences between STAS and TAS and identify causes of performance degradation.
• To introduce an effective end-to-end model for STAS applicable to online video analysis.

Main Methods:

• Developed a novel end-to-end streaming video TAS model with reinforcement learning (SVTAS-RL).
• Utilized reinforcement learning (RL) to overcome optimization dilemmas inherent in online learning.
• Designed the model to mitigate bias arising from the shift from offline to online task nature.

Main Results:

• The SVTAS-RL model significantly outperforms existing STAS approaches.
• Achieved competitive performance compared to state-of-the-art TAS models on multiple datasets.
• Demonstrated notable advantages on the ultralong video dataset EGTEA, validating its effectiveness for long-form content.

Conclusions:

• The SVTAS-RL model effectively addresses the challenges of streaming temporal action segmentation.
• End-to-end modeling and reinforcement learning are key to successful online video analysis.
• The proposed method offers a viable solution for real-time video understanding applications.