Recurrent Flow Update Model Using Image Pyramid Structure for 4K Video Frame Interpolation

Area of Science:

• Computer Vision
• Image Processing
• Artificial Intelligence

Background:

• Video frame interpolation (VFI) synthesizes intermediate frames between existing ones.
• Current VFI methods, pixel-level synthesis and flow-based approaches, face challenges with high-resolution video and accurate flow estimation.
• Separate training of multi-stage models often leads to suboptimal VFI results.

Purpose of the Study:

• To develop an improved VFI method that overcomes limitations of existing approaches for high-resolution video synthesis.
• To propose a novel Recurrent Flow Update (RFU) model trained end-to-end.
• To enhance the accuracy and efficiency of video frame generation.

Main Methods:

• Introduced a Recurrent Flow Update (RFU) model for end-to-end training.
• Developed a global flow update module to leverage global information and correct flow errors.
• Utilized ablation studies to validate the effectiveness of the proposed method.

Main Results:

• The RFU model achieved state-of-the-art performance on 4K resolution datasets (XTest, Davis).
• The method also demonstrated superior results on the SNU-FILM dataset, which includes large motions.
• The end-to-end training approach and global flow update module proved effective in mitigating VFI challenges.

Conclusions:

• The proposed Recurrent Flow Update (RFU) model offers a significant advancement in video frame interpolation.
• End-to-end training and the global flow update module effectively address limitations in existing VFI techniques.
• The method achieves state-of-the-art results for high-resolution and large-motion video synthesis.