CLIP-RL: Closed-Loop Video Inpainting with Detection-Guided Reinforcement Learning

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Existing video inpainting methods often fail to adapt to diverse scenarios, leading to temporal inconsistencies and quality degradation.
• These methods struggle with capturing high-level temporal semantics crucial for realistic video restoration.

Purpose of the Study:

• To introduce a novel reinforcement learning framework for adaptive video inpainting strategy optimization.
• To address limitations in current methods regarding temporal consistency and semantic understanding.

Main Methods:

• Reformulated video inpainting as an agent-environment interaction within a closed-loop framework (CLIP-RL).
• Employed a policy network and a composite reward function with temporal alignment loss for adaptive strategy selection.
• Utilized a pre-trained inpainting detection module for real-time quality feedback.

Main Results:

• CLIP-RL improved Peak Signal-to-Noise Ratio (PSNR) from 34.43 to 34.67 and Structural Similarity Index Measure (SSIM) from 0.974 to 0.986 on the YouTube-VOS dataset compared to ProPainter.
• Demonstrated superior performance in detail preservation and artifact suppression through qualitative analysis.

Conclusions:

• CLIP-RL offers an effective adaptive strategy optimization for video inpainting.
• The reinforcement learning approach enhances temporal consistency and overall visual quality in video restoration.