BiSTNet: Semantic Image Prior Guided Bidirectional Temporal Feature Fusion for Deep Exemplar-Based Video Colorization

Area of Science:

• Computer Vision
• Artificial Intelligence
• Image Processing

Background:

• Exemplar-based video colorization requires effective color exploration and propagation.
• Existing methods often struggle with color bleeding and inaccurate alignment.

Purpose of the Study:

• To present BiSTNet, a novel network for exemplar-based video colorization.
• To improve color propagation accuracy and mitigate artifacts using semantic priors.

Main Methods:

• Establishing semantic correspondence between frames and exemplars in deep feature space.
• Employing bidirectional temporal feature fusion for color propagation.
• Utilizing a mixed expert block for object boundary modeling and a multi-scale refinement block for progressive colorization.

Main Results:

• BiSTNet demonstrates superior performance compared to state-of-the-art methods on benchmark datasets.
• The method successfully reduces color bleeding artifacts around object boundaries.
• Achieved champion performance in the NTIRE 2023 video colorization challenge.

Conclusions:

• BiSTNet offers an effective solution for exemplar-based video colorization.
• The integration of semantic priors and advanced fusion techniques enhances colorization quality.
• The proposed method sets a new benchmark in video colorization tasks.