Progressive multi-branch video style transfer network via confidence reweighted projection

Achieving temporal consistency in video content poses a significant challenge for high-quality video styling. Unfortunately, current video style transfer techniques often amplify the differences between frames in the input video, leading to unsmooth transitions between the stylized video frames. This can lead to visual flickering and discontinuities over time. Besides, the lack of style consistency between video frames or between different modes can make the overall style of the video appear fragmented or unnatural. To address these issues, this paper introduces an innovative real-time, end-to-end model for video style transfer,named as Progressive Multi-Branch Video Style Transfer Network (PMBNet). The primary focus is on introducing a Multi-layer Branch Semantic Transformation Structure (MBSTS), which comprises a multi-scale feature extraction layer featuring a multi-parallel progressive sub-modules (MPP), as well as a multi-level feature fusion layer that includes multiple channel spatial attention sub-modules (CSA). Among them, the MPP sub-modules use progressively dilated convolutions to enlarge the receptive field and mitigate grid artifacts, enabling the model to capture temporal and spatial dependencies, improving temporal consistency and reducing flickering between frames. The CSA sub-modules fuse content and style features across branches, ensuring multi-scale extraction and consistent style application, which resolves style fragmentation and ensures smooth transitions and consistency across frames and video modes. Additionally, a confidence reweighted calculation is employed to choose a dominant pattern from several potential options, ensuring consistency in modal content structure and preserving perceptual quality. Comprehensive evaluation demonstrates that PMBNet surpasses the performance of current state-of-the-art methods.