Video Frame Interpolation via Appearance-Based Intermediate Flow Estimation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Intermediate flow estimation is crucial for video frame interpolation (VFI).
• Existing methods often fail with extreme motion due to assumptions of localized linear motion.
• The effectiveness of VFI is limited by its ability to accurately capture complex object movements.

Purpose of the Study:

• To develop a new intermediate flow estimation method for VFI that addresses limitations with extreme motion.
• To improve VFI performance by linking object motion trajectories to their visual appearance characteristics.
• To enhance inter-frame feature extraction for more robust motion estimation.

Main Methods:

• Propose a novel intermediate flow estimation method based on the assumption that object motion is determined by appearance.
• Extract motion features from image appearance and inter-frame motion.
• Utilize a modified Swin-Transformer with cyclically shifted windows for adaptive neighborhood feature extraction.

Main Results:

• The proposed method achieves state-of-the-art performance on various datasets for both fixed-time and arbitrary-time interpolation.
• Demonstrates superior performance compared to existing models, especially in handling videos with extreme motion.
• Outperforms methods requiring four input frames when dealing with significant object displacement.

Conclusions:

• The proposed appearance-driven intermediate flow estimation method significantly enhances video frame interpolation, particularly for challenging scenarios with extreme motion.
• Rethinking feature extraction with adaptive neighborhoods and Swin-Transformer modifications improves VFI accuracy.
• This approach offers a more robust solution for generating high-quality intermediate frames.