Exploiting Optical Flow Guidance for Transformer-Based Video Inpainting.

Area of Science:

• Computer Vision
• Artificial Intelligence

Background:

• Transformers, utilizing multi-head self-attention (MHSA), are prevalent in video processing.
• Video inpainting faces challenges due to degraded features in corrupted regions, leading to inaccurate self-attention (query degradation).
• Previous work (FGT) showed optical flow guidance can mitigate query degradation.

Purpose of the Study:

• To develop a more effective and efficient video inpainting method by enhancing flow guidance.
• To address the query degradation problem in transformer-based video inpainting.

Main Methods:

• Designed a lightweight flow completion network with local aggregation and edge loss.
• Introduced a flow guidance feature integration module to enhance features using motion discrepancy.
• Developed a flow-guided feature propagation module for feature warping.
• Decoupled transformer into temporal and spatial dimensions with temporally deformable and dual-perspective MHSA mechanisms guided by optical flows.

Main Results:

• FGT++ effectively addresses query degradation in video inpainting.
• The proposed modules enhance feature representation and propagation.
• Decoupled transformer architecture improves temporal and spatial attention mechanisms.

Conclusions:

• FGT++ demonstrates state-of-the-art performance in video inpainting.
• The enhanced flow guidance strategy significantly improves inpainting quality and efficiency.
• This work offers a promising direction for robust video restoration tasks.