Quantifying and Learning Static vs. Dynamic Information in Deep Spatiotemporal Networks.

Area of Science:

• Computer Vision
• Deep Learning
• Artificial Intelligence

Background:

• Deep spatiotemporal models lack clear understanding of their intermediate representations.
• Quantitative methods for evaluating static vs. dynamic bias in these models are absent.

Purpose of the Study:

• To propose and apply a quantitative methodology for assessing static and dynamic biases in spatiotemporal models.
• To analyze these biases across action recognition, automatic video object segmentation (AVOS), and video instance segmentation (VIS).

Main Methods:

• Developed a novel approach to quantify static and dynamic information biases within spatiotemporal models.
• Applied the methodology to analyze multiple deep learning architectures across various video understanding tasks.
• Introduced StaticDropout for action recognition to reduce static bias and enhance dynamic information utilization.

Main Results:

• Identified a prevalent static bias in most examined spatiotemporal models.
• Found that some datasets presumed dynamic-biased actually exhibit static bias.
• Observed that individual model channels can specialize in static, dynamic, or combined information.
• Determined that model biases stabilize within the first half of training.

Conclusions:

• Spatiotemporal models often over-rely on static visual information, necessitating bias quantification.
• The proposed methods and StaticDropout offer avenues for improving model performance on dynamic tasks.
• Architectural insights reveal channel-level specialization for static and dynamic features.