DyDiT++: Diffusion Transformers With Timestep and Spatial Dynamics for Efficient Visual Generation.

Area of Science:

• Computer Vision
• Artificial Intelligence
• Machine Learning

Background:

• Diffusion models, particularly the Diffusion Transformer (DiT), excel at visual generation but incur high computational expenses.
• Static inference in current models leads to redundant computations across timesteps and spatial regions.

Purpose of the Study:

• To develop a Dynamic Diffusion Transformer (DyDiT) that optimizes computational efficiency in visual generation.
• To introduce methods for dynamic adjustment of computation along both temporal and spatial dimensions.

Main Methods:

• Proposed Timestep-wise Dynamic Width (TDW) to adapt model width based on generation timesteps.
• Introduced Spatial-wise Dynamic Token (SDT) to eliminate redundant computations in unnecessary spatial areas.
• Developed DyDiT++ incorporating flow matching, video generation, text-to-image generation, and parameter-efficient training (TD-LoRA).

Main Results:

• DyDiT significantly accelerates the generation process by reducing redundant computations.
• DyDiT++ demonstrates versatility by accelerating flow-matching generation and enhancing complex visual tasks.
• Parameter-efficient fine-tuning with TD-LoRA achieves significant FLOP reduction (<3% fine-tuning iterations) and hardware speedup (1.73x) for DiT XL, with competitive FID scores (2.07 on ImageNet).

Conclusions:

• DyDiT and DyDiT++ offer substantial computational savings and speedups for diffusion-based visual generation.
• The dynamic approaches enhance efficiency across various visual generation tasks and model architectures.
• Parameter-efficient training democratizes access to advanced visual generation technologies.