MotionPrior: Exploring Efficient Learning of Motion Concepts for Few-Shot Video Generation

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Diffusion models have advanced text-to-image generation but video generation remains resource-intensive.
• Existing few-shot video generation methods are limited to single motion concepts.
• There's a need for cost-effective video generation with improved flexibility and reduced data/resource demands.

Purpose of the Study:

• To develop a cost-effective video generation scheme capable of handling adaptive and multiple motion concepts.
• To enhance generation freedom in video synthesis without a significant increase in training burden.
• To leverage limited video data for learning motion priors.

Main Methods:

• A learnable bank for motion concepts was constructed.
• A Dual-Semantic-guided Motion Attention module was proposed to extract motion elements using textual and visual guidance.
• Lightweight motion injection layers, a temporal-aware noise prior, and inter-frame consistency constraints were employed.

Main Results:

• The proposed method successfully learns motion priors adaptively from small datasets.
• Generated videos exhibit smooth motion and support single or multiple motion concepts.
• Experimental results show superior performance compared to existing few-shot and some large-scale video generation models.

Conclusions:

• The developed scheme offers a cost-effective solution for flexible video generation.
• It effectively integrates motion semantics with reduced parameters and computational cost.
• The approach demonstrates the potential for improved generation freedom in text-to-video synthesis using limited resources.