Vidsketch: Hand-drawn sketch-driven video generation with diffusion control

Area of Science:

• Computer Vision
• Artificial Intelligence
• Generative Models

Background:

• Existing generative AI methods excel at static image generation from sketches.
• Current sketch-guided video generation is limited, often requiring advanced drawing skills via edge maps.
• There's a need for accessible tools enabling video animation synthesis from simple sketches.

Purpose of the Study:

• To introduce VidSketch, a pioneering method for generating high-quality video animations from hand-drawn sketches and text prompts.
• To lower the barrier to entry for video content creation, bridging the gap between amateur and professional users.
• To provide a flexible and user-friendly approach to sketch-guided video animation.

Main Methods:

• Developed a Sketch Interpolation Strategy to expand user-provided sketch sequences into full video frames.
• Introduced a Level-Based Sketch Control Strategy to dynamically adjust sketch guidance based on user proficiency.
• Integrated text prompts for enhanced control over video content generation.

Main Results:

• VidSketch demonstrates superior performance and generalizability compared to baseline methods on VBench metrics.
• The method successfully generates high-quality video animations from diverse sketch inputs.
• Experimental results validate the effectiveness of the proposed interpolation and control strategies.

Conclusions:

• VidSketch represents a significant advancement in sketch-guided video animation synthesis.
• The approach enhances input flexibility and accommodates varying user drawing skills.
• VidSketch empowers a wider range of users to create video content effectively.