Multi-Modality Self-Distillation for Weakly Supervised Temporal Action Localization.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Weakly-supervised Temporal Action Localization (WTAL) is crucial for high-level video understanding but struggles with precise action boundary detection due to whole-video classification labels.
• Existing pseudo-label methods for WTAL often fail to fully leverage multi-modal data (RGB and optical flow) and effectively mitigate label noise, limiting feature representation learning.

Purpose of the Study:

• To propose a novel Multi-Modality Self-Distillation (MMSD) framework to address the limitations of current WTAL methods.
• To enhance the accuracy of action instance boundary detection in videos using multi-modal information.
• To improve the robustness of WTAL models by mitigating label noise through a self-voting mechanism.

Main Methods:

• The proposed MMSD framework utilizes two single-modal streams (RGB and optical flow) and a fused-modal stream.
• It incorporates multi-modality knowledge distillation to transfer information between streams, enhancing snippet-level classification.
• Multi-modality self-voting is employed to reduce label noise by considering the reliability and complementarity of different modalities.

Main Results:

• The MMSD framework demonstrated significant improvements in weakly-supervised temporal action localization.
• Experimental results on THUMOS14 and ActivityNet1.3 datasets show superior performance compared to existing state-of-the-art approaches.
• The method effectively mitigates label noise and enhances discriminative feature representation learning.

Conclusions:

• The Multi-Modality Self-Distillation (MMSD) framework offers an effective solution for WTAL by leveraging multi-modal data and robust label noise reduction.
• The proposed approach achieves state-of-the-art performance, highlighting the benefits of knowledge distillation and self-voting in multi-modal settings.
• The code is publicly available, facilitating further research and development in temporal action localization.