Test-Time Adaptation for Optical Flow Estimation Using Motion Vectors.

Area of Science:

• Computer Vision
• Machine Learning
• Deep Learning

Background:

• Deep learning models for optical flow estimation often struggle with real-world data due to reliance on synthetic training data.
• Performance degradation occurs because of distribution shifts between synthetic training and real-world testing environments.

Purpose of the Study:

• To develop a method for adapting optical flow estimation models at test time, enhancing their generalization to real-world scenarios.
• To address the challenges posed by the high cost and technical difficulties of annotating real-world optical flow data.

Main Methods:

• Propose a self-supervised learning task for adapting optical flow models during test time.
• Utilize motion vectors and residuals readily available from compressed video formats.
• Formulate the self-supervised task as motion vector prediction, linking it to optical flow estimation.

Main Results:

• The proposed Test-Time Adaptation guided with Motion Vectors (TTA-MV) is the first to adapt optical flow estimation using motion vectors.
• TTA-MV significantly improves the generalization capabilities of established deep learning optical flow methods.
• Experimental results show enhanced performance for models like FlowNet, PWCNet, and RAFT when using TTA-MV.

Conclusions:

• TTA-MV offers an effective solution for bridging the gap between synthetic training and real-world performance in optical flow estimation.
• The methodology provides a practical approach to enhance the robustness of optical flow models without requiring ground-truth annotations.
• This work paves the way for more reliable and widely applicable deep learning-based optical flow solutions.