Vidsketch: Hand-drawn sketch-driven video generation with diffusion control

With the advancement of generative artificial intelligence, previous studies have achieved the task of generating aesthetic images from hand-drawn sketches, fulfilling the public's needs for drawing. However, these methods are limited to static image generation and cannot control video animation synthesis via hand-drawn sketches. Moreover, the few sketch-guided video approaches often rely on concrete edge maps, placing high demands on users' drawing skills and raising the barrier to entry. To address these two key challenges, we propose VidSketch-an early attempt at one of the first methods capable of generating high-quality video animations directly from any number of hand-drawn sketches and simple text prompts. VidSketch bridges the gap between amateur users and professional artists in video content creation. Specifically, we design a Sketch Interpolation Strategy that expands any user-provided sketch sequence into a full set of video frames, offering greater input flexibility. Furthermore, we introduce a Level-Based Sketch Control Strategy that dynamically adjusts sketch guidance strength during generation to accommodate users with varying drawing proficiency. Experimental results show that VidSketch outperforms baseline methods on most VBench metrics, demonstrating superior performance and generalizability. Additional examples of generated animations are available at https://csfufu.github.io/vid_sketch/.