Study of Spatio-Temporal Modeling in Video Quality Assessment.

Area of Science:

• Computer Vision
• Machine Learning
• Signal Processing

Background:

• Video quality assessment (VQA) is crucial for evaluating video content.
• Recurrent Neural Networks (RNNs) are commonly used in VQA to model temporal dynamics.
• Current VQA models face challenges in effectively learning long-term temporal quality variations.

Purpose of the Study:

• To investigate the actual role of RNNs in learning video quality.
• To determine if RNNs learn spatio-temporal representations or redundantly aggregate spatial features.
• To explore the impact of frame sampling strategies and spatio-temporal fusion on VQA performance.

Main Methods:

• Trained a family of VQA models using diverse frame sampling strategies.
• Implemented various spatio-temporal fusion techniques.
• Conducted extensive experiments on four public in-the-wild video quality datasets.

Main Results:

• The spatio-temporal modeling module (RNNs) did not enhance quality-aware spatio-temporal feature learning.
• Sparsely sampled video frames achieved performance comparable to using all frames.
• Spatial features were found to be critical for capturing video quality variations.

Conclusions:

• RNNs do not significantly contribute to learning spatio-temporal features for VQA.
• Effective VQA can be achieved with sparse frame sampling, emphasizing spatial information.
• This study pioneers the exploration of spatio-temporal modeling limitations in VQA.