Consistent and Controllable Image Animation With Linear Motion Diffusion Transformers.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Diffusion models have advanced image animation, but challenges remain in appearance consistency and motion smoothness.
• Current image animation methods often use U-Net diffusion models, which are less advanced than Transformer-based text-to-video (T2V) models.
• Vanilla self-attention in Transformers leads to high computational costs, limiting efficiency in image animation.

Purpose of the Study:

• To introduce MiraMo, a framework for efficient, consistent, and smooth image animation.
• To address the limitations of existing diffusion models in appearance consistency and motion control.
• To improve the computational efficiency of image animation generation.

Main Methods:

• Employed a text-to-video architecture with efficient linear attention instead of vanilla self-attention.
• Introduced a motion residual learning paradigm to model motion dynamics for better temporal consistency.
• Utilized a DCT-based noise refinement strategy and a dynamics control module for artifact suppression and motion control.

Main Results:

• MiraMo demonstrates superior performance in generating consistent and smooth animations compared to state-of-the-art methods.
• The framework achieves accelerated inference speeds, reducing computational overhead.
• Experiments validate MiraMo's effectiveness in motion transfer and video editing tasks.

Conclusions:

• MiraMo offers an efficient and effective solution for high-quality image animation.
• The framework successfully improves appearance consistency, motion smoothness, and controllability.
• MiraMo represents a significant advancement in diffusion-based image animation techniques.