Exploring the Temporal Consistency of Arbitrary Style Transfer: A Channelwise Perspective

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Neural style transfer is popular for images, but extending it to videos causes flickering.
• Existing methods struggle with temporal consistency, leading to artifacts like flickering in stylized videos.

Purpose of the Study:

• To analyze the cause of flickering in neural video stylization.
• To propose a novel method that ensures temporal consistency and reduces flickering.

Main Methods:

• Analyzed feature migration modules in state-of-the-art (SOTA) systems, identifying channelwise misalignment as a cause of flickering.
• Developed a multichannel correlation network (MCCNet) for direct frame alignment in feature space.
• Introduced an inner channel similarity loss and an illumination loss to enhance alignment and performance.

Main Results:

• MCCNet effectively aligns output frames with input frames in the hidden feature space.
• The proposed losses mitigate side effects and improve performance under varying light conditions.
• Evaluations show MCCNet achieves high performance in arbitrary video and image style transfer.

Conclusions:

• MCCNet successfully addresses the temporal inconsistency and flickering issues in neural video stylization.
• The method offers a simple yet efficient solution for high-quality arbitrary video style transfer.
• This work provides a robust approach for generating temporally consistent stylized videos.