Learning Self-Supervised Space-Time CNN for Fast Video Style Transfer.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Deep Learning

Background:

• Image style transfer has advanced with Convolutional Neural Networks (CNNs).
• Independent frame-by-frame video style transfer causes flickering and instability.
• Real-time, temporally coherent video style transfer remains a challenge.

Purpose of the Study:

• To develop a self-supervised, end-to-end deep learning method for online video style transfer.
• To achieve real-time stylized video generation with temporal coherence.
• To address the flickering issue in existing video style transfer methods.

Main Methods:

• Proposed VTNet, a space-time CNN for online video style transfer.
• Employed a temporal prediction branch for spatiotemporal features and a stylizing branch for style transfer.
• Introduced a Style-Coherence Loss Net (SCNet) with content, style, and coherence losses, using a pretrained VGG-16 network.
• Designed a novel coherence loss to ensure temporal consistency without explicit optical flow.

Main Results:

• VTNet successfully transfers image style to video frames in real-time.
• The method produces temporally coherent stylized videos, significantly reducing flickering.
• Evaluations show favorable results compared to state-of-the-art methods in efficiency and quality.

Conclusions:

• VTNet offers an efficient and effective solution for real-time, temporally coherent video style transfer.
• The self-supervised, end-to-end approach trained on unlabeled data demonstrates strong performance.
• The proposed coherence loss is key to achieving stable and natural-looking stylized videos.